Facial Emotion Detection

Problem Definition

The context: Why is this problem important to solve?

As technology becomes more integrated into daily life, enhancing human-computer interaction is essential. Facial emotion detection allows computers to understand and respond to human emotions, making interactions more natural and intuitive. It can also helps solving problems unaffected by mood or getting tired, which is important at safety critical tasks. Some examples of applications are:

• Mental Health Monitoring by detecting changes in emotional states over time, it could provide early warnings of mental health issues such as depression or anxiety, enabling timely intervention.
• Enhanced User Experience in Various sectors like retail, entertainment, and marketing, understanding customer emotions that may enhance user experience.
• Automotive Safety where emotion detection can monitor the driver’s state, detecting signs of fatigue or stress, and potentially preventing accidents.
• Security Systems where facial emotion detection can be used to detect suspicious behavior or stress, which could be indicative of criminal intentions or distress situations.
• Healthcare and Patient Care by understanding patient emotions, which can improve care quality.
• Robotics and AI where, emotional intelligence is key for creating more sophisticated, empathetic, and interactive robots.

The objectives: What is the intended goal?

After an initial assessment involving a visual validation on the image dataset to be used as training, validation and testing input for the Machine Learning models, a comparison will be made between various pre-trained models and custom models, with the goal of identifying which model performs best for this particular task of Object Classification of Facial Emotions.

The key questions: What are the key questions that need to be answered?

Some of the key questions to be answered are:

• Data Quality and Consistency: Is the dataset suitable or have any inconsistencies that could affect the training and evaluation of models?
• Model Suitability: Which pre-trained models are most suitable for this task, and how do they compare to custom models developed specifically for facial emotion recognition?
• Performance Metrics: Beyond accuracy, what other metrics (like precision, recall, F1 score) are important for evaluating the performance of these models in this context?
• Generalization Ability: How well do these models generalize to unseen data, are there any signs of overfitting?
• Computational Efficiency: What are the trade-offs between model accuracy and computational resources required, such as training time?

The problem formulation: What are we trying to solve using data science?

By utilising Machine Learning and AI we can accelerate and process data in a more deterministic and much faster rate than humans might, or with significantly less resources. Some of the other issues that we are trying to solve are:

• Optimizing Human-Computer Interaction: By integrating facial emotion detection into computer systems, we aim to create more responsive and intuitive interactions, making technology more attuned to human needs and emotional states.

• Enhancing Safety and Security: Through the detection of emotional cues, such as stress or fatigue, particularly in critical sectors like automotive and security systems, we aim to proactively address safety concerns and potentially prevent accidents or security breaches.

• Improving Healthcare and Mental Health Monitoring: By identifying emotional changes over time, we can offer better insights into mental health and patient care. This could lead to early detection of conditions like depression or anxiety, and improve the overall quality of care in healthcare settings.

• Advancing Retail, Entertainment, and Marketing: Understanding customer emotions in real-time can revolutionize these sectors by providing enhanced user experiences, personalized content, and improved customer service.

• Developing Emotional Intelligence in Robotics and AI: The goal is to endow robots and AI systems with the ability to recognize and respond to human emotions, paving the way for more empathetic and sophisticated interactions between humans and machines.

With Data Science, especifically Machine Learning and AI we can not only accurately interpret facial emotions but also apply this understanding in various real-world applications to improve safety, healthcare, user experience, and human-computer interactions, amongst many other applications.

About the dataset

The data set consists of 3 folders, i.e., 'test', 'train', and 'validation'. Each of these folders has four subfolders:

‘happy’: Images of people who have happy facial expressions.
‘sad’: Images of people with sad or upset facial expressions.
‘surprise’: Images of people who have shocked or surprised facial expressions.
‘neutral’: Images of people showing no prominent emotion in their facial expression at all.

Mounting the Drive

NOTE: Please use Google Colab from your browser for this notebook. Google.colab is NOT a library that can be downloaded locally on your device.

from google.colab import drive
drive.mount('/content/drive')

Mounted at /content/drive

Importing the Libraries

import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler

import tensorflow as tf
from tensorflow.keras.preprocessing.image import load_img, img_to_array
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.layers import Dense, Input, Dropout, GlobalAveragePooling2D, Flatten, Conv2D, BatchNormalization, Activation, MaxPooling2D, LeakyReLU
from tensorflow.keras.models import Model, Sequential

import zipfile

#Checking Tensorflow version
print(tf.__version__)

2.15.0

Let us load and unzip the data

Note:

• You must download the dataset from the link provided on Olympus and upload the same on your Google drive before executing the code in the next cell.
• In case of any error, please make sure that the path of the file is correct as the path may be different for you.

# Loading dataset to variable 'path'
path = '/content/drive/MyDrive/Useful Docs/Facial_emotion_images.zip'

# Extracting dataset from zipfile
with zipfile.ZipFile(path, 'r') as zip_ref:
    zip_ref.extractall()

# Loading dataset to variable 'path' if working locally
path = '/Users/ceb/Downloads/TEMP/Facial_emotion_images'

import os

folder_path = "Facial_emotion_images/"
emotions_data = {}

for root, dirs, files in os.walk(folder_path):
    for dir_name in dirs:
        # Split the root to get the main folder and subfolder names
        parts = root.split(os.sep)
        if len(parts) > 1:
            main_folder = parts[-1]
            sub_folder = dir_name

            # Initialize nested dictionary if not already present
            if main_folder not in emotions_data:
                emotions_data[main_folder] = {}

            # Initialize the sub-folder dictionary
            dir_path = os.path.join(root, dir_name)
            emotions_data[main_folder][sub_folder] = {'path': dir_path, 'count': 0}

from PIL import Image
import os

folder_path = "Facial_emotion_images/"
emotions_data = {}

# Initialize Nested Emotion Data Dictionary
for root, dirs, files in os.walk(folder_path):
    for dir_name in dirs:
        parts = root.split(os.sep)
        if len(parts) > 1:
            main_folder = parts[-1]
            sub_folder = dir_name

            if main_folder not in emotions_data:
                emotions_data[main_folder] = {}

            dir_path = os.path.join(root, dir_name)
            emotions_data[main_folder][sub_folder] = {'path': dir_path, 'count': 0, 'size': None}

# Count Files and Determine Size
for main_folder in emotions_data:
    for sub_folder in emotions_data[main_folder]:
        dir_path = emotions_data[main_folder][sub_folder]['path']
        file_count = 0

        if os.path.isdir(dir_path):
            for file in os.listdir(dir_path):
                if file.lower().endswith(('.png', '.jpg', '.jpeg')):
                    if file_count == 0:  # Check size for the first image
                        with Image.open(os.path.join(dir_path, file)) as img:
                            emotions_data[main_folder][sub_folder]['size'] = img.size
                    file_count += 1

        emotions_data[main_folder][sub_folder]['count'] = file_count

        # Print the updated information
        print(f"Folder: {main_folder}, Emotion: {sub_folder}, Count: {emotions_data[main_folder][sub_folder]['count']}, Size: {emotions_data[main_folder][sub_folder]['size']}, Path: {dir_path}")

Folder: , Emotion: test, Count: 0, Size: None, Path: Facial_emotion_images/test
Folder: , Emotion: train, Count: 0, Size: None, Path: Facial_emotion_images/train
Folder: , Emotion: validation, Count: 0, Size: None, Path: Facial_emotion_images/validation
Folder: test, Emotion: happy, Count: 32, Size: (48, 48), Path: Facial_emotion_images/test/happy
Folder: test, Emotion: sad, Count: 32, Size: (48, 48), Path: Facial_emotion_images/test/sad
Folder: test, Emotion: surprise, Count: 32, Size: (48, 48), Path: Facial_emotion_images/test/surprise
Folder: test, Emotion: neutral, Count: 32, Size: (48, 48), Path: Facial_emotion_images/test/neutral
Folder: train, Emotion: happy, Count: 3976, Size: (48, 48), Path: Facial_emotion_images/train/happy
Folder: train, Emotion: sad, Count: 3982, Size: (48, 48), Path: Facial_emotion_images/train/sad
Folder: train, Emotion: surprise, Count: 3173, Size: (48, 48), Path: Facial_emotion_images/train/surprise
Folder: train, Emotion: neutral, Count: 3978, Size: (48, 48), Path: Facial_emotion_images/train/neutral
Folder: validation, Emotion: happy, Count: 1825, Size: (48, 48), Path: Facial_emotion_images/validation/happy
Folder: validation, Emotion: sad, Count: 1139, Size: (48, 48), Path: Facial_emotion_images/validation/sad
Folder: validation, Emotion: surprise, Count: 797, Size: (48, 48), Path: Facial_emotion_images/validation/surprise
Folder: validation, Emotion: neutral, Count: 1216, Size: (48, 48), Path: Facial_emotion_images/validation/neutral

# Confirming whether the structure works, it can be reused later in the code
print(emotions_data['train']['sad']['path'])
print(emotions_data['train']['sad']['size'])
print(emotions_data['train']['sad']['count'])

Facial_emotion_images/train/sad
(48, 48)
3982

# Store image size to be reused since all images are 48x48 pixels
img_size = emotions_data['train']['sad']['size']

Visualizing our Classes

Happy

# Reusing variables from earlier data exploration
emotion = 'happy'
path = emotions_data['train'][emotion]['path']

# Plotting images for visualisation
plt.figure(figsize= (8,8))
for i in range(1, 10, 1):
  plt.subplot(3, 3, i)
  img = load_img(path + "/" +
                os.listdir(path)[i], target_size = (img_size))
  plt.imshow(img)
plt.show()

Observations and Insights:The 'happy' images predominantly feature smiles, with the corners of the mouth turned upwards and eyes often crinkling. A key distinguishing feature is the genuine warmth in the eyes, setting it apart from neutral or sad expressions. However, some challenging images include subtle smiles, which could be confused with a neutral expression, especially in lower-resolution or unclear images. Some images are also partially ocluded which could be a challenge for the Neural Network.

Sad

# Reusing variables from earlier data exploration
emotion = 'sad'
path = emotions_data['train'][emotion]['path']

# Plotting images for visualisation
plt.figure(figsize= (8,8))
for i in range(1, 10, 1):
  plt.subplot(3, 3, i)
  img = load_img(path + "/" +
                os.listdir(path)[i], target_size = (img_size))
  plt.imshow(img)
plt.show()

Observations and Insights: Sad expressions are characterized by downturned mouths, furrowed brows, and sometimes teary eyes. A unique aspect is the overall downward trajectory of facial features. Challenges arise in distinguishing subtle sad expressions from neutral ones, as some individuals may exhibit less pronounced sadness cues.

Neutral

# Reusing variables from earlier data exploration
emotion = 'neutral'
path = emotions_data['train'][emotion]['path']

# Plotting images for visualisation
plt.figure(figsize= (8,8))
for i in range(1, 10, 1):
  plt.subplot(3, 3, i)
  img = load_img(path + "/" +
                os.listdir(path)[i], target_size = (img_size))
  plt.imshow(img)
plt.show()

Observations and Insights: Neutral expressions are marked by a lack of significant emotional indicators. The mouth is typically relaxed and straight, and the eyes are open without notable crinkling or squinting. The challenge in this category is differentiating between subtle emotional expressions and truly neutral ones, as minimal cues can sometimes be misinterpreted.

Surprised

# Reusing variables from earlier data exploration
emotion = 'surprise'
path = emotions_data['train'][emotion]['path']

# Plotting images for visualisation
plt.figure(figsize= (8,8))
for i in range(1, 10, 1):
  plt.subplot(3, 3, i)
  img = load_img(path + "/" +
                os.listdir(path)[i], target_size = (img_size))
  plt.imshow(img)
plt.show()

Observations and Insights: Surprised expressions are easily identifiable by wide-open eyes and often an open mouth. The uniqueness lies in the sense of suddenness and exaggeration in the features. The difficulty is in differentiating between positive surprise and other high-arousal emotions, like happy, in some images where context and lesser cues are present.

Checking Distribution of Classes

# 'category' will be 'train', 'test' and 'validation'
for category in emotions_data:
  for emotion in emotions_data[category]:
    if emotions_data[category][emotion]['count'] != 0:
      print("The {} dataset has {} {} images.".format(category, emotions_data[category][emotion]['count'], emotion))

The test dataset has 32 happy images.
The test dataset has 32 sad images.
The test dataset has 32 surprise images.
The test dataset has 32 neutral images.
The train dataset has 3976 happy images.
The train dataset has 3982 sad images.
The train dataset has 3173 surprise images.
The train dataset has 3978 neutral images.
The validation dataset has 1825 happy images.
The validation dataset has 1139 sad images.
The validation dataset has 797 surprise images.
The validation dataset has 1216 neutral images.

# Calculating the ratio between largest dataset and smallest. Other datasets sizes are similar to 'sad' dataset
surprise_ratio = emotions_data['train']['surprise']['count']/emotions_data['train']['sad']['count']
print("The 'surprise' dataset is {:.1f}% smaller than 'sad' dataset".format((1-surprise_ratio)*100))

The 'surprise' dataset is 20.3% smaller than 'sad' dataset

Further Data Exploration

Visualising Validation Dataset for any obvious issues on the images

# Visualising the validation and test datasets
emotion = 'happy'
path = emotions_data['validation'][emotion]['path']

# Plotting images for visualisation
plt.figure(figsize= (8,8))
for i in range(1, 10, 1):
  plt.subplot(3, 3, i)
  img = load_img(path + "/" +
                os.listdir(path)[i], target_size = (img_size))
  plt.imshow(img)
plt.show()

# Visualising the validation and test datasets
emotion = 'sad'
path = emotions_data['validation'][emotion]['path']

# Plotting images for visualisation
plt.figure(figsize= (8,8))
for i in range(1, 10, 1):
  plt.subplot(3, 3, i)
  img = load_img(path + "/" +
                os.listdir(path)[i], target_size = (img_size))
  plt.imshow(img)
plt.show()

# Visualising the validation and test datasets
emotion = 'neutral'
path = emotions_data['validation'][emotion]['path']

# Plotting images for visualisation
plt.figure(figsize= (8,8))
for i in range(1, 10, 1):
  plt.subplot(3, 3, i)
  img = load_img(path + "/" +
                os.listdir(path)[i], target_size = (img_size))
  plt.imshow(img)
plt.show()

# Visualising the validation and test datasets
emotion = 'surprise'
path = emotions_data['validation'][emotion]['path']

# Plotting images for visualisation
plt.figure(figsize= (8,8))
for i in range(1, 10, 1):
  plt.subplot(3, 3, i)
  img = load_img(path + "/" +
                os.listdir(path)[i], target_size = (img_size))
  plt.imshow(img)
plt.show()

Visualising Test Dataset for any obvious issues on the images

# Visualising the validation and test datasets
emotion = 'happy'
path = emotions_data['test'][emotion]['path']

# Plotting images for visualisation
plt.figure(figsize= (8,8))
for i in range(1, 10, 1):
  plt.subplot(3, 3, i)
  img = load_img(path + "/" +
                os.listdir(path)[i], target_size = (img_size))
  plt.imshow(img)
plt.show()

# Visualising the validation and test datasets
emotion = 'sad'
path = emotions_data['test'][emotion]['path']

# Plotting images for visualisation
plt.figure(figsize= (8,8))
for i in range(1, 10, 1):
  plt.subplot(3, 3, i)
  img = load_img(path + "/" +
                os.listdir(path)[i], target_size = (img_size))
  plt.imshow(img)
plt.show()

# Visualising the validation and test datasets
emotion = 'neutral'
path = emotions_data['test'][emotion]['path']

# Plotting images for visualisation
plt.figure(figsize= (8,8))
for i in range(1, 10, 1):
  plt.subplot(3, 3, i)
  img = load_img(path + "/" +
                os.listdir(path)[i], target_size = (img_size))
  plt.imshow(img)
plt.show()

# Visualising the validation and test datasets
emotion = 'surprise'
path = emotions_data['test'][emotion]['path']

# Plotting images for visualisation
plt.figure(figsize= (8,8))
for i in range(1, 10, 1):
  plt.subplot(3, 3, i)
  img = load_img(path + "/" +
                os.listdir(path)[i], target_size = (img_size))
  plt.imshow(img)
plt.show()

# Checking image sizes
print("Happy images have the size of", emotions_data['train']['happy']['size'])
print("Happy images have the size of", emotions_data['train']['sad']['size'])
print("Happy images have the size of", emotions_data['train']['neutral']['size'])
print("Happy images have the size of", emotions_data['train']['surprise']['size'])

Happy images have the size of (48, 48)
Happy images have the size of (48, 48)
Happy images have the size of (48, 48)
Happy images have the size of (48, 48)

Further Data Exploration has revealed:

• Images are 48x48 pixels in size
• Test dataset has exactly 32 images for each emotion
• Validation dataset has a significant imbalance with many more happy images and a significantly lower amount of surprise images. This over/under representation will most likely affect the result of the training.
• Some images contain watermarks (e.g. Shutterstock)

Observations and Insights: The noted imbalance in the 'surprise' dataset, being significantly smaller than the 'sad' dataset (20.3% smaller), indicates a skew in the data. The test dataset containing 32 happy images also suggests potential imbalances. While some degree of class imbalance is common in real-world datasets, the extent of this discrepancy might pose challenges.

• Class Distribution: The classes are not equally distributed. The 20.3% smaller 'surprise' dataset compared to 'sad' suggests a notable imbalance. Such disparities can impact the model's performance, potentially leading it to be less accurate in identifying less-represented emotions.

• Impact of Imbalance: This imbalance could be problematic as it may cause the neural network to be biased towards recognizing emotions that are more frequently represented in the training data. It may underperform in identifying 'surprise' due to the lack of representation in the data.

Creating our Data Loaders

In this section, we are creating data loaders that we will use as inputs to our Neural Network.

# Batch size to be used for training
batch_size  = 32

# Create an instance of ImageDataGenerator
datagen = ImageDataGenerator(horizontal_flip = True,
          brightness_range=(0.,2.),
          rescale=1./255,
          shear_range=0.3)

# Creating dataloader for train for RGB dataset
train_set_rgb = datagen.flow_from_directory(folder_path + "train",
                                        target_size = (img_size),
                                        color_mode = 'rgb',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        shuffle = True)
# Creating dataloader for train for Grayscale dataset
train_set_gray = datagen.flow_from_directory(folder_path + "train",
                                        target_size = (img_size),
                                        color_mode = 'grayscale',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        shuffle = True)

# Creating dataloader for validation for RGB dataset
validation_set_rgb = datagen.flow_from_directory(folder_path + "validation",
                                        target_size = (img_size),
                                        color_mode = 'rgb',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        shuffle = True)
# Creating dataloader for validation for Grayscale dataset
validation_set_gray = datagen.flow_from_directory(folder_path + "validation",
                                        target_size = (img_size),
                                        color_mode = 'grayscale',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        shuffle = True)

# Creating dataloader for test for RGB dataset
test_set_rgb = datagen.flow_from_directory(folder_path + "test",
                                        target_size = (img_size),
                                        color_mode = 'rgb',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        shuffle = True)
# Creating dataloader for test for Grayscale dataset
test_set_gray = datagen.flow_from_directory(folder_path + "test",
                                        target_size = (img_size),
                                        color_mode = 'grayscale',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        shuffle = True)

Found 15109 images belonging to 4 classes.
Found 15109 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.
Found 128 images belonging to 4 classes.
Found 128 images belonging to 4 classes.

Model Building

# Fixing the seed for random number generators
fixed_seed = 42

#Creating a function as we'll be resetting the seed multiple times throughout the notebook
def reset_seed(seed):
  np.random.seed(seed)
  import random
  random.seed(seed)
  tf.random.set_seed(seed)
  print("Random number generators seed has been set to", seed)

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Activation, BatchNormalization, Flatten, Conv2D, MaxPooling2D, LeakyReLU
from tensorflow.keras.losses import categorical_crossentropy
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.optimizers import legacy  
from tensorflow.keras.utils import to_categorical
from tensorflow.keras import backend

# Clearing backend
backend.clear_session()

# Reset seed for number generators using reset_seed() function
reset_seed(fixed_seed)

Random number generators seed has been set to 42

Creating the Base Neural Network

# Defining Base NN
def cnn_model_1(colour_layers):
    cnn_model = Sequential()

    # First Convolutional layer with 64 filters and the kernel size of 3x3, 'same' padding and input shape = (48, 48, 3 - RGB colours)
    cnn_model.add(Conv2D(filters = 64, kernel_size = (2, 2), padding = "same", input_shape = (img_size[0], img_size[1], colour_layers), activation = 'relu'))
    # A max-pooling layer with a pool size of 2x2
    cnn_model.add(MaxPooling2D(pool_size = (2, 2)))
    # Dropout layer with the rate equal to 0.2
    cnn_model.add(Dropout(0.2))
    # Second Convolutional layer with 32 filters and the kernel size of 2x2 with 'same' padding
    cnn_model.add(Conv2D(filters = 32, kernel_size = (2, 2), padding = "same", activation = 'relu'))
    # A max-pooling layer with a pool size of 2x2
    cnn_model.add(MaxPooling2D(pool_size = (2, 2)))
    # Dropout layer with the rate equal to 0.2
    cnn_model.add(Dropout(0.2))
    # Third Convolutional layer with 32 filters and the kernel size of 2x2 with 'same' padding
    cnn_model.add(Conv2D(filters = 32, kernel_size = (2, 2), padding = "same", activation = 'relu'))
    # A max-pooling layer with a pool size of 2x2
    cnn_model.add(MaxPooling2D(pool_size = (2, 2)))
    # Dropout layer with the rate equal to 0.2
    cnn_model.add(Dropout(0.2))
    # Flatten the output from the previous layer
    cnn_model.add(Flatten())
    # Dense layer with 512 nodes
    cnn_model.add(Dense(512, activation = 'relu'))
    # Dropout layer with the rate equal to 0.4
    cnn_model.add(Dropout(0.4))
    # Final output layer with nodes equal to the number of classes (sad, happy, neutral and surprised) and 'softmax' as the activation function
    cnn_model.add(Dense(4, activation = 'softmax'))
    # Compiling the CNN model with categorical crossentropy as loss function, Adam Optimizer
    # with 0.001 learning rate, and set metrics set to 'accuracy'.
    cnn_model.compile(loss='categorical_crossentropy',
              optimizer=legacy.Adam(learning_rate = 0.001),
              metrics=['accuracy'])
    return cnn_model

Compiling and Training the Model

from keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau

# ModelCheckpoint is used to save the model at certain intervals
checkpoint = ModelCheckpoint("model1.h5",     # file path where the model will be saved
                             monitor='val_accuracy', # means the callback monitors the validation accuracy
                             verbose=1,         # model is saved only when the metric (val_acc) has improved
                             save_best_only=True,
                             mode='max')

# Stop the training process early if a monitored metric has stopped improving
early_stopping = EarlyStopping(monitor = 'val_loss',  # monitors the validation loss
                          min_delta = 0,              # any positive change is considered as an improvement
                          patience = 3,               # training will be stopped if no improvement is seen for 3 epochs
                          verbose = 1,
                          restore_best_weights = True # weights are reverted to best value after early stopping
                          )

# Reduces the learning rate when a metric has stopped improving
reduce_learningrate = ReduceLROnPlateau(monitor = 'val_loss', # monitor the validation loss
                              factor = 0.2,                   # learning rate reduced to 20% when reduction is triggered
                              patience = 3,                   # reduction happens after 3 epochs with no improvement
                              verbose = 1,
                              min_delta = 0.0001)

# Configured callbacks will be passed to the model during training
callbacks_list = [early_stopping, checkpoint, reduce_learningrate]

# Number of iterations over the entire dataset that the training process should run
epochs = 20

# Building the model
model1 = cnn_model_1(3)
# Print first model's summary
model1.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 64)        832       
                                                                 
 max_pooling2d (MaxPooling2  (None, 24, 24, 64)        0         
 D)                                                              
                                                                 
 dropout (Dropout)           (None, 24, 24, 64)        0         
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 32)        8224      
                                                                 
 max_pooling2d_1 (MaxPoolin  (None, 12, 12, 32)        0         
 g2D)                                                            
                                                                 
 dropout_1 (Dropout)         (None, 12, 12, 32)        0         
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 32)        4128      
                                                                 
 max_pooling2d_2 (MaxPoolin  (None, 6, 6, 32)          0         
 g2D)                                                            
                                                                 
 dropout_2 (Dropout)         (None, 6, 6, 32)          0         
                                                                 
 flatten (Flatten)           (None, 1152)              0         
                                                                 
 dense (Dense)               (None, 512)               590336    
                                                                 
 dropout_3 (Dropout)         (None, 512)               0         
                                                                 
 dense_1 (Dense)             (None, 4)                 2052      
                                                                 
=================================================================
Total params: 605572 (2.31 MB)
Trainable params: 605572 (2.31 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________

# Visualize the model as blocks
from tensorflow.keras.utils import plot_model
plot_model(model1, to_file='model.png', show_shapes=True, show_layer_names=True)

# Fitting the model with required parameters above, saving model into history variable
history = model1.fit(
    train_set_rgb,
    steps_per_epoch = train_set_rgb.samples // train_set_rgb.batch_size,
    validation_data = validation_set_rgb,
    validation_steps = validation_set_rgb.samples // validation_set_rgb.batch_size,
    epochs = 20,
    verbose = 1,
    callbacks = callbacks_list
)

Epoch 1/20
472/472 [==============================] - ETA: 0s - loss: 1.3347 - accuracy: 0.3343
Epoch 1: val_accuracy improved from -inf to 0.46310, saving model to model1.h5
472/472 [==============================] - 15s 32ms/step - loss: 1.3347 - accuracy: 0.3343 - val_loss: 1.2183 - val_accuracy: 0.4631 - lr: 0.0010
Epoch 2/20
  5/472 [..............................] - ETA: 11s - loss: 1.1861 - accuracy: 0.5312

/Users/ceb/anaconda3/lib/python3.11/site-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

472/472 [==============================] - ETA: 0s - loss: 1.1885 - accuracy: 0.4742
Epoch 2: val_accuracy improved from 0.46310 to 0.49859, saving model to model1.h5
472/472 [==============================] - 15s 31ms/step - loss: 1.1885 - accuracy: 0.4742 - val_loss: 1.1562 - val_accuracy: 0.4986 - lr: 0.0010
Epoch 3/20
472/472 [==============================] - ETA: 0s - loss: 1.1079 - accuracy: 0.5114
Epoch 3: val_accuracy improved from 0.49859 to 0.57802, saving model to model1.h5
472/472 [==============================] - 15s 32ms/step - loss: 1.1079 - accuracy: 0.5114 - val_loss: 1.0088 - val_accuracy: 0.5780 - lr: 0.0010
Epoch 4/20
471/472 [============================>.] - ETA: 0s - loss: 1.0537 - accuracy: 0.5370
Epoch 4: val_accuracy did not improve from 0.57802
472/472 [==============================] - 16s 33ms/step - loss: 1.0536 - accuracy: 0.5370 - val_loss: 0.9875 - val_accuracy: 0.5712 - lr: 0.0010
Epoch 5/20
471/472 [============================>.] - ETA: 0s - loss: 1.0129 - accuracy: 0.5593
Epoch 5: val_accuracy improved from 0.57802 to 0.59637, saving model to model1.h5
472/472 [==============================] - 15s 32ms/step - loss: 1.0128 - accuracy: 0.5591 - val_loss: 0.9513 - val_accuracy: 0.5964 - lr: 0.0010
Epoch 6/20
470/472 [============================>.] - ETA: 0s - loss: 0.9952 - accuracy: 0.5742
Epoch 6: val_accuracy improved from 0.59637 to 0.59859, saving model to model1.h5
472/472 [==============================] - 15s 32ms/step - loss: 0.9952 - accuracy: 0.5744 - val_loss: 0.9347 - val_accuracy: 0.5986 - lr: 0.0010
Epoch 7/20
471/472 [============================>.] - ETA: 0s - loss: 0.9745 - accuracy: 0.5829
Epoch 7: val_accuracy improved from 0.59859 to 0.61613, saving model to model1.h5
472/472 [==============================] - 15s 32ms/step - loss: 0.9744 - accuracy: 0.5826 - val_loss: 0.9284 - val_accuracy: 0.6161 - lr: 0.0010
Epoch 8/20
471/472 [============================>.] - ETA: 0s - loss: 0.9541 - accuracy: 0.5880
Epoch 8: val_accuracy did not improve from 0.61613
472/472 [==============================] - 15s 32ms/step - loss: 0.9537 - accuracy: 0.5882 - val_loss: 0.9179 - val_accuracy: 0.6093 - lr: 0.0010
Epoch 9/20
472/472 [==============================] - ETA: 0s - loss: 0.9440 - accuracy: 0.5959
Epoch 9: val_accuracy did not improve from 0.61613
472/472 [==============================] - 15s 31ms/step - loss: 0.9440 - accuracy: 0.5959 - val_loss: 0.9320 - val_accuracy: 0.6153 - lr: 0.0010
Epoch 10/20
471/472 [============================>.] - ETA: 0s - loss: 0.9281 - accuracy: 0.6023
Epoch 10: val_accuracy improved from 0.61613 to 0.63690, saving model to model1.h5
472/472 [==============================] - 15s 32ms/step - loss: 0.9274 - accuracy: 0.6026 - val_loss: 0.8957 - val_accuracy: 0.6369 - lr: 0.0010
Epoch 11/20
470/472 [============================>.] - ETA: 0s - loss: 0.9142 - accuracy: 0.6110
Epoch 11: val_accuracy improved from 0.63690 to 0.64214, saving model to model1.h5
472/472 [==============================] - 15s 32ms/step - loss: 0.9138 - accuracy: 0.6109 - val_loss: 0.8720 - val_accuracy: 0.6421 - lr: 0.0010
Epoch 12/20
471/472 [============================>.] - ETA: 0s - loss: 0.9040 - accuracy: 0.6149
Epoch 12: val_accuracy improved from 0.64214 to 0.64778, saving model to model1.h5
472/472 [==============================] - 15s 32ms/step - loss: 0.9044 - accuracy: 0.6144 - val_loss: 0.8334 - val_accuracy: 0.6478 - lr: 0.0010
Epoch 13/20
472/472 [==============================] - ETA: 0s - loss: 0.8861 - accuracy: 0.6217
Epoch 13: val_accuracy improved from 0.64778 to 0.65202, saving model to model1.h5
472/472 [==============================] - 15s 32ms/step - loss: 0.8861 - accuracy: 0.6217 - val_loss: 0.8418 - val_accuracy: 0.6520 - lr: 0.0010
Epoch 14/20
471/472 [============================>.] - ETA: 0s - loss: 0.8841 - accuracy: 0.6239
Epoch 14: val_accuracy did not improve from 0.65202
472/472 [==============================] - 15s 33ms/step - loss: 0.8843 - accuracy: 0.6239 - val_loss: 0.8714 - val_accuracy: 0.6433 - lr: 0.0010
Epoch 15/20
472/472 [==============================] - ETA: 0s - loss: 0.8712 - accuracy: 0.6328Restoring model weights from the end of the best epoch: 12.

Epoch 15: val_accuracy did not improve from 0.65202

Epoch 15: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026.
472/472 [==============================] - 15s 32ms/step - loss: 0.8712 - accuracy: 0.6328 - val_loss: 0.8391 - val_accuracy: 0.6510 - lr: 0.0010
Epoch 15: early stopping

Evaluating the Model on the Test Set

from keras.models import load_model

# Load the saved model if notebook session gets disconnected
model1 = load_model("model1.h5")

# Evaluate the model on the test data using
test_loss, test_accuracy = model1.evaluate(test_set_rgb, verbose=1)

# Print the results
print("Test Loss:", test_loss)
print("Test Accuracy:", test_accuracy)

4/4 [==============================] - 0s 24ms/step - loss: 0.8422 - accuracy: 0.6094
Test Loss: 0.8422161936759949
Test Accuracy: 0.609375

# Importing relevant libraries
import seaborn as sns
import matplotlib.pyplot as plt

# Plotting the Training and Validation accuracies of CNN model 2
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('CNN Model 1 Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc = 'upper left')

# Display the plot
plt.show()

Trying out with Greyscale dataset

# Clearing backend
backend.clear_session()

# Reset seed for number generators using reset_seed() function
reset_seed(fixed_seed)

Random number generators seed has been set to 42

# ModelCheckpoint is used to save the model at certain intervals
checkpoint_grayscale1 = ModelCheckpoint("model1_grayscale.h5",     # file path where the model will be saved
                             monitor='val_accuracy', # means the callback monitors the validation accuracy
                             verbose=1,         # model is saved only when the metric (val_acc) has improved
                             save_best_only=True,
                             mode='max')

# Configured callbacks will be passed to the model during training
callbacks_list_grayscale1 = [early_stopping, checkpoint_grayscale1, reduce_learningrate]

# Number of iterations over the entire dataset that the training process should run
epochs = 20

# Building the model
model1_gray = cnn_model_1(1)
# Print first model's summary
model1_gray.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 64)        320       
                                                                 
 max_pooling2d (MaxPooling2  (None, 24, 24, 64)        0         
 D)                                                              
                                                                 
 dropout (Dropout)           (None, 24, 24, 64)        0         
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 32)        8224      
                                                                 
 max_pooling2d_1 (MaxPoolin  (None, 12, 12, 32)        0         
 g2D)                                                            
                                                                 
 dropout_1 (Dropout)         (None, 12, 12, 32)        0         
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 32)        4128      
                                                                 
 max_pooling2d_2 (MaxPoolin  (None, 6, 6, 32)          0         
 g2D)                                                            
                                                                 
 dropout_2 (Dropout)         (None, 6, 6, 32)          0         
                                                                 
 flatten (Flatten)           (None, 1152)              0         
                                                                 
 dense (Dense)               (None, 512)               590336    
                                                                 
 dropout_3 (Dropout)         (None, 512)               0         
                                                                 
 dense_1 (Dense)             (None, 4)                 2052      
                                                                 
=================================================================
Total params: 605060 (2.31 MB)
Trainable params: 605060 (2.31 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________

# Fitting the model with required parameters above, saving model into history variable
history_grayscale1 = model1_gray.fit(
    train_set_gray,
    steps_per_epoch = train_set_gray.samples // train_set_gray.batch_size,
    validation_data = validation_set_gray,
    validation_steps = validation_set_gray.samples // validation_set_gray.batch_size,
    epochs = 20,
    verbose = 1,
    callbacks = callbacks_list_grayscale1
)

Epoch 1/20
472/472 [==============================] - ETA: 0s - loss: 1.3379 - accuracy: 0.3391
Epoch 1: val_accuracy improved from -inf to 0.46875, saving model to model1_grayscale.h5
472/472 [==============================] - 13s 27ms/step - loss: 1.3379 - accuracy: 0.3391 - val_loss: 1.2111 - val_accuracy: 0.4688 - lr: 0.0010
Epoch 2/20
  5/472 [..............................] - ETA: 14s - loss: 1.2863 - accuracy: 0.4000

/Users/ceb/anaconda3/lib/python3.11/site-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

471/472 [============================>.] - ETA: 0s - loss: 1.2135 - accuracy: 0.4542
Epoch 2: val_accuracy improved from 0.46875 to 0.51552, saving model to model1_grayscale.h5
472/472 [==============================] - 12s 26ms/step - loss: 1.2135 - accuracy: 0.4541 - val_loss: 1.1344 - val_accuracy: 0.5155 - lr: 0.0010
Epoch 3/20
471/472 [============================>.] - ETA: 0s - loss: 1.1304 - accuracy: 0.4964
Epoch 3: val_accuracy improved from 0.51552 to 0.55625, saving model to model1_grayscale.h5
472/472 [==============================] - 13s 27ms/step - loss: 1.1303 - accuracy: 0.4964 - val_loss: 1.0474 - val_accuracy: 0.5562 - lr: 0.0010
Epoch 4/20
472/472 [==============================] - ETA: 0s - loss: 1.0900 - accuracy: 0.5176
Epoch 4: val_accuracy did not improve from 0.55625
472/472 [==============================] - 12s 25ms/step - loss: 1.0900 - accuracy: 0.5176 - val_loss: 1.0598 - val_accuracy: 0.5482 - lr: 0.0010
Epoch 5/20
472/472 [==============================] - ETA: 0s - loss: 1.0431 - accuracy: 0.5460
Epoch 5: val_accuracy improved from 0.55625 to 0.58266, saving model to model1_grayscale.h5
472/472 [==============================] - 12s 25ms/step - loss: 1.0431 - accuracy: 0.5460 - val_loss: 0.9935 - val_accuracy: 0.5827 - lr: 0.0010
Epoch 6/20
471/472 [============================>.] - ETA: 0s - loss: 1.0222 - accuracy: 0.5595
Epoch 6: val_accuracy improved from 0.58266 to 0.59294, saving model to model1_grayscale.h5
472/472 [==============================] - 12s 25ms/step - loss: 1.0220 - accuracy: 0.5596 - val_loss: 0.9528 - val_accuracy: 0.5929 - lr: 0.0010
Epoch 7/20
472/472 [==============================] - ETA: 0s - loss: 1.0013 - accuracy: 0.5666
Epoch 7: val_accuracy improved from 0.59294 to 0.60302, saving model to model1_grayscale.h5
472/472 [==============================] - 12s 25ms/step - loss: 1.0013 - accuracy: 0.5666 - val_loss: 0.9354 - val_accuracy: 0.6030 - lr: 0.0010
Epoch 8/20
470/472 [============================>.] - ETA: 0s - loss: 0.9904 - accuracy: 0.5714
Epoch 8: val_accuracy improved from 0.60302 to 0.62097, saving model to model1_grayscale.h5
472/472 [==============================] - 12s 26ms/step - loss: 0.9907 - accuracy: 0.5716 - val_loss: 0.9110 - val_accuracy: 0.6210 - lr: 0.0010
Epoch 9/20
470/472 [============================>.] - ETA: 0s - loss: 0.9641 - accuracy: 0.5796
Epoch 9: val_accuracy improved from 0.62097 to 0.63044, saving model to model1_grayscale.h5
472/472 [==============================] - 12s 25ms/step - loss: 0.9638 - accuracy: 0.5800 - val_loss: 0.9019 - val_accuracy: 0.6304 - lr: 0.0010
Epoch 10/20
471/472 [============================>.] - ETA: 0s - loss: 0.9505 - accuracy: 0.5922
Epoch 10: val_accuracy did not improve from 0.63044
472/472 [==============================] - 12s 25ms/step - loss: 0.9511 - accuracy: 0.5918 - val_loss: 0.9001 - val_accuracy: 0.6282 - lr: 0.0010
Epoch 11/20
470/472 [============================>.] - ETA: 0s - loss: 0.9368 - accuracy: 0.5989
Epoch 11: val_accuracy did not improve from 0.63044
472/472 [==============================] - 12s 26ms/step - loss: 0.9366 - accuracy: 0.5990 - val_loss: 0.9057 - val_accuracy: 0.6232 - lr: 0.0010
Epoch 12/20
471/472 [============================>.] - ETA: 0s - loss: 0.9254 - accuracy: 0.6050
Epoch 12: val_accuracy improved from 0.63044 to 0.64375, saving model to model1_grayscale.h5
472/472 [==============================] - 12s 26ms/step - loss: 0.9249 - accuracy: 0.6051 - val_loss: 0.8743 - val_accuracy: 0.6438 - lr: 0.0010
Epoch 13/20
470/472 [============================>.] - ETA: 0s - loss: 0.9142 - accuracy: 0.6088
Epoch 13: val_accuracy did not improve from 0.64375
472/472 [==============================] - 12s 26ms/step - loss: 0.9144 - accuracy: 0.6086 - val_loss: 0.8682 - val_accuracy: 0.6371 - lr: 0.0010
Epoch 14/20
472/472 [==============================] - ETA: 0s - loss: 0.9004 - accuracy: 0.6109
Epoch 14: val_accuracy did not improve from 0.64375
472/472 [==============================] - 12s 25ms/step - loss: 0.9004 - accuracy: 0.6109 - val_loss: 0.8820 - val_accuracy: 0.6345 - lr: 0.0010
Epoch 15/20
471/472 [============================>.] - ETA: 0s - loss: 0.8977 - accuracy: 0.6205
Epoch 15: val_accuracy did not improve from 0.64375
472/472 [==============================] - 12s 26ms/step - loss: 0.8980 - accuracy: 0.6203 - val_loss: 0.8789 - val_accuracy: 0.6389 - lr: 0.0010
Epoch 16/20
472/472 [==============================] - ETA: 0s - loss: 0.8903 - accuracy: 0.6174
Epoch 16: val_accuracy improved from 0.64375 to 0.65363, saving model to model1_grayscale.h5
472/472 [==============================] - 12s 26ms/step - loss: 0.8903 - accuracy: 0.6174 - val_loss: 0.8336 - val_accuracy: 0.6536 - lr: 0.0010
Epoch 17/20
471/472 [============================>.] - ETA: 0s - loss: 0.8807 - accuracy: 0.6275
Epoch 17: val_accuracy did not improve from 0.65363
472/472 [==============================] - 12s 26ms/step - loss: 0.8805 - accuracy: 0.6276 - val_loss: 0.8538 - val_accuracy: 0.6456 - lr: 0.0010
Epoch 18/20
472/472 [==============================] - ETA: 0s - loss: 0.8693 - accuracy: 0.6273
Epoch 18: val_accuracy improved from 0.65363 to 0.66331, saving model to model1_grayscale.h5
472/472 [==============================] - 12s 26ms/step - loss: 0.8693 - accuracy: 0.6273 - val_loss: 0.8141 - val_accuracy: 0.6633 - lr: 0.0010
Epoch 19/20
472/472 [==============================] - ETA: 0s - loss: 0.8609 - accuracy: 0.6359
Epoch 19: val_accuracy did not improve from 0.66331
472/472 [==============================] - 13s 27ms/step - loss: 0.8609 - accuracy: 0.6359 - val_loss: 0.8188 - val_accuracy: 0.6569 - lr: 0.0010
Epoch 20/20
472/472 [==============================] - ETA: 0s - loss: 0.8411 - accuracy: 0.6495
Epoch 20: val_accuracy improved from 0.66331 to 0.66573, saving model to model1_grayscale.h5
472/472 [==============================] - 13s 27ms/step - loss: 0.8411 - accuracy: 0.6495 - val_loss: 0.8293 - val_accuracy: 0.6657 - lr: 0.0010

# Load the saved model if notebook session gets disconnected
model1_gray = load_model("model1_grayscale.h5")

# Evaluate the model on the test data using Grayscale
test_loss, test_accuracy = model1_gray.evaluate(test_set_gray, verbose=1)

# Print the results
print("Test Loss:", test_loss)
print("Test Accuracy:", test_accuracy)

4/4 [==============================] - 0s 15ms/step - loss: 0.8388 - accuracy: 0.5938
Test Loss: 0.838821530342102
Test Accuracy: 0.59375

# Plotting the Training and Validation accuracies of CNN model 1 and Grayscale images
plt.plot(history_grayscale1.history['accuracy'])
plt.plot(history_grayscale1.history['val_accuracy'])
plt.title('CNN Model 1 Accuracy with Grayscale Images')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc = 'upper left')

# Display the plot
plt.show()

Observations and Insights:

• The Greyscale model has performed slightly better than the RGB model in training and validation, with a slightly faster processing time per epoch as well.
  • RGB: 63% training accuracy, 65% validation accuracy.
  • Greyscale: 64.9% training accuracy, 66.5% validation accuracy.
• The use of Convolutional Neural Networks (CNNs) is the correct approach vs Artificial Neural Networks (ANNs). For image-based tasks, CNNs are more suitable than ANNs, because CNNs are specifically designed to process pixel data and can capture spatial hierarchies in images, which ANNs cannot efficiently do.
• Given that the images in the dataset are inherently grayscale, using an RGB model may not have been the most efficient approach. RGB models are designed to process and learn from color information, which is absent in this case. This could partly explain why the grayscale model outperformed the RGB model.
• The grayscale model's higher accuracy (66.57% in validation) as compared to the RGB model (65.10% in validation) indicates that it is more suited to this specific dataset. Perhaps due to images being originally greyscale, the model can focus on learning and extracting relevant features directly related to intensity and texture without the distraction of non-existent color information.
• Processing grayscale images requires less computational power and memory compared to RGB images. This makes the grayscale model more resource-efficient, which is particularly advantageous when working with large datasets or limited computational resources.
• Further experiments and model optimizations should focus on grayscale models. Additionally, exploring various architectures and hyperparameters specifically tailored for grayscale image processing could yield further improvements in accuracy.
• The accuracy plot shows that the models haven't platoed in terms of efficiency improvement over the epochs. Perhaps relaxing the conditions at which Tensorflow stops the training could have allowed the model to continue to improve over time.

Creating the second Convolutional Neural Network

• Try out a slightly larger architecture

# Clearing backend
backend.clear_session()

# Reset seed for number generators using reset_seed() function
reset_seed(fixed_seed)

Random number generators seed has been set to 42

# Define more complex CNN model
def cnn_model_2(colour_layers):
    cnn_model = Sequential()

    # First Convolutional layer with 256 filters and the kernel size of 3x3, 'same' padding and input shape = (48, 48, 3 - RGB colours)
    cnn_model.add(Conv2D(filters = 256, kernel_size = (2, 2), padding = "same", input_shape = (img_size[0], img_size[1], colour_layers), activation = 'relu'))
    # Adding a BatchNormalization layer
    cnn_model.add(BatchNormalization())
    # LeakyRelU layer with Leaky ReLU parameter of 0.1
    cnn_model.add(LeakyReLU(0.1))
    # A max-pooling layer with a pool size of 2x2
    cnn_model.add(MaxPooling2D(pool_size = (2, 2)))
    # Second Convolutional layer with 128 filters and the kernel size of 2x2 with 'same' padding
    cnn_model.add(Conv2D(filters = 128, kernel_size = (2, 2), padding = "same", activation = 'relu'))
    # Adding a BatchNormalization layer
    cnn_model.add(BatchNormalization())
    # LeakyRelU layer with Leaky ReLU parameter of 0.1
    cnn_model.add(LeakyReLU(0.1))
    # A max-pooling layer with a pool size of 2x2
    cnn_model.add(MaxPooling2D(pool_size = (2, 2)))
    # Third Convolutional layer with 32 filters and the kernel size of 2x2 with 'same' padding
    cnn_model.add(Conv2D(filters = 64, kernel_size = (2, 2), padding = "same", activation = 'relu'))
    # Adding a BatchNormalization layer
    cnn_model.add(BatchNormalization())
    # LeakyRelU layer with Leaky ReLU parameter of 0.1
    cnn_model.add(LeakyReLU(0.1))
    # A max-pooling layer with a pool size of 2x2
    cnn_model.add(MaxPooling2D(pool_size = (2, 2)))
    # Fourth Convolutional layer with 32 filters and the kernel size of 2x2 with 'same' padding
    cnn_model.add(Conv2D(filters = 32, kernel_size = (2, 2), padding = "same", activation = 'relu'))
    # Adding a BatchNormalization layer
    cnn_model.add(BatchNormalization())
    # LeakyRelU layer with Leaky ReLU parameter of 0.1
    cnn_model.add(LeakyReLU(0.1))
    # A max-pooling layer with a pool size of 2x2
    cnn_model.add(MaxPooling2D(pool_size = (2, 2)))
    # Flatten the output from the previous layer
    cnn_model.add(Flatten())
    # Dense layer with 512 nodes
    cnn_model.add(Dense(512, activation = 'relu'))
    # Dense layer with 128 nodes
    cnn_model.add(Dense(128, activation = 'relu'))
    # Dropout layer with the rate equal to 0.4
    cnn_model.add(Dropout(0.4))
    # Final output layer with nodes equal to the number of classes (sad, happy, neutral and surprised) and 'softmax' as the activation function
    cnn_model.add(Dense(4, activation = 'softmax'))
    # Compiling the CNN model with categorical crossentropy as loss function, Adam Optimizer
    # with 0.001 learning rate, and set metrics set to 'accuracy'.
    cnn_model.compile(loss='categorical_crossentropy',
              optimizer=legacy.Adam(learning_rate = 0.001),
              metrics=['accuracy'])
    return cnn_model

Compiling and Training the Model

# Building the model
model2_rgb = cnn_model_2(3)
# Print first model's summary
model2_rgb.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 256)       3328      
                                                                 
 batch_normalization (Batch  (None, 48, 48, 256)       1024      
 Normalization)                                                  
                                                                 
 leaky_re_lu (LeakyReLU)     (None, 48, 48, 256)       0         
                                                                 
 max_pooling2d (MaxPooling2  (None, 24, 24, 256)       0         
 D)                                                              
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 128)       131200    
                                                                 
 batch_normalization_1 (Bat  (None, 24, 24, 128)       512       
 chNormalization)                                                
                                                                 
 leaky_re_lu_1 (LeakyReLU)   (None, 24, 24, 128)       0         
                                                                 
 max_pooling2d_1 (MaxPoolin  (None, 12, 12, 128)       0         
 g2D)                                                            
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 64)        32832     
                                                                 
 batch_normalization_2 (Bat  (None, 12, 12, 64)        256       
 chNormalization)                                                
                                                                 
 leaky_re_lu_2 (LeakyReLU)   (None, 12, 12, 64)        0         
                                                                 
 max_pooling2d_2 (MaxPoolin  (None, 6, 6, 64)          0         
 g2D)                                                            
                                                                 
 conv2d_3 (Conv2D)           (None, 6, 6, 32)          8224      
                                                                 
 batch_normalization_3 (Bat  (None, 6, 6, 32)          128       
 chNormalization)                                                
                                                                 
 leaky_re_lu_3 (LeakyReLU)   (None, 6, 6, 32)          0         
                                                                 
 max_pooling2d_3 (MaxPoolin  (None, 3, 3, 32)          0         
 g2D)                                                            
                                                                 
 flatten (Flatten)           (None, 288)               0         
                                                                 
 dense (Dense)               (None, 512)               147968    
                                                                 
 dense_1 (Dense)             (None, 128)               65664     
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_2 (Dense)             (None, 4)                 516       
                                                                 
=================================================================
Total params: 391652 (1.49 MB)
Trainable params: 390692 (1.49 MB)
Non-trainable params: 960 (3.75 KB)
_________________________________________________________________

# Visualize the model as blocks
plot_model(model2_rgb, to_file='model.png', show_shapes=True, show_layer_names=True)

# ModelCheckpoint is used to save the model at certain intervals
checkpoint_model2_rgb = ModelCheckpoint("model2_rgb.h5", # file path where the model will be saved
                             monitor='val_accuracy',  # means the callback monitors the validation accuracy
                             verbose=1,               # model is saved only when the metric (val_acc) has improved
                             save_best_only=True,
                             mode='max')

# Configured callbacks will be passed to the model during training
callbacks_list_model2_rgb = [early_stopping, checkpoint_model2_rgb, reduce_learningrate]

# Number of iterations over the entire dataset that the training process should run
epochs = 20

# Fitting the model with required parameters above, saving model into history_model2 variable
history_model2_rgb = model2_rgb.fit(
    train_set_rgb,
    steps_per_epoch = train_set_rgb.samples // train_set_rgb.batch_size,
    validation_data = validation_set_rgb,
    validation_steps = validation_set_rgb.samples // validation_set_rgb.batch_size,
    epochs = 20,
    verbose = 1,
    callbacks = callbacks_list_model2_rgb
)

Epoch 1/20
472/472 [==============================] - ETA: 0s - loss: 1.3467 - accuracy: 0.3299
Epoch 1: val_accuracy improved from -inf to 0.37036, saving model to model2_rgb.h5
472/472 [==============================] - 83s 175ms/step - loss: 1.3467 - accuracy: 0.3299 - val_loss: 1.3109 - val_accuracy: 0.3704 - lr: 0.0010
Epoch 2/20
472/472 [==============================] - ETA: 0s - loss: 1.2211 - accuracy: 0.4125
Epoch 2: val_accuracy improved from 0.37036 to 0.42702, saving model to model2_rgb.h5
472/472 [==============================] - 81s 172ms/step - loss: 1.2211 - accuracy: 0.4125 - val_loss: 1.1961 - val_accuracy: 0.4270 - lr: 0.0010
Epoch 3/20
472/472 [==============================] - ETA: 0s - loss: 1.1291 - accuracy: 0.4863
Epoch 3: val_accuracy improved from 0.42702 to 0.47520, saving model to model2_rgb.h5
472/472 [==============================] - 82s 173ms/step - loss: 1.1291 - accuracy: 0.4863 - val_loss: 1.1530 - val_accuracy: 0.4752 - lr: 0.0010
Epoch 4/20
472/472 [==============================] - ETA: 0s - loss: 1.0326 - accuracy: 0.5530
Epoch 4: val_accuracy improved from 0.47520 to 0.55867, saving model to model2_rgb.h5
472/472 [==============================] - 82s 173ms/step - loss: 1.0326 - accuracy: 0.5530 - val_loss: 1.0323 - val_accuracy: 0.5587 - lr: 0.0010
Epoch 5/20
472/472 [==============================] - ETA: 0s - loss: 0.9555 - accuracy: 0.5907
Epoch 5: val_accuracy improved from 0.55867 to 0.60565, saving model to model2_rgb.h5
472/472 [==============================] - 82s 173ms/step - loss: 0.9555 - accuracy: 0.5907 - val_loss: 0.9409 - val_accuracy: 0.6056 - lr: 0.0010
Epoch 6/20
472/472 [==============================] - ETA: 0s - loss: 0.9048 - accuracy: 0.6131
Epoch 6: val_accuracy did not improve from 0.60565
472/472 [==============================] - 81s 172ms/step - loss: 0.9048 - accuracy: 0.6131 - val_loss: 0.9495 - val_accuracy: 0.5954 - lr: 0.0010
Epoch 7/20
472/472 [==============================] - ETA: 0s - loss: 0.8587 - accuracy: 0.6407
Epoch 7: val_accuracy improved from 0.60565 to 0.61935, saving model to model2_rgb.h5
472/472 [==============================] - 82s 174ms/step - loss: 0.8587 - accuracy: 0.6407 - val_loss: 0.9314 - val_accuracy: 0.6194 - lr: 0.0010
Epoch 8/20
472/472 [==============================] - ETA: 0s - loss: 0.8298 - accuracy: 0.6519
Epoch 8: val_accuracy improved from 0.61935 to 0.62883, saving model to model2_rgb.h5
472/472 [==============================] - 84s 177ms/step - loss: 0.8298 - accuracy: 0.6519 - val_loss: 0.8953 - val_accuracy: 0.6288 - lr: 0.0010
Epoch 9/20
472/472 [==============================] - ETA: 0s - loss: 0.8065 - accuracy: 0.6612
Epoch 9: val_accuracy improved from 0.62883 to 0.65000, saving model to model2_rgb.h5
472/472 [==============================] - 83s 175ms/step - loss: 0.8065 - accuracy: 0.6612 - val_loss: 0.8580 - val_accuracy: 0.6500 - lr: 0.0010
Epoch 10/20
472/472 [==============================] - ETA: 0s - loss: 0.7860 - accuracy: 0.6741
Epoch 10: val_accuracy improved from 0.65000 to 0.65887, saving model to model2_rgb.h5
472/472 [==============================] - 82s 174ms/step - loss: 0.7860 - accuracy: 0.6741 - val_loss: 0.8170 - val_accuracy: 0.6589 - lr: 0.0010
Epoch 11/20
472/472 [==============================] - ETA: 0s - loss: 0.7630 - accuracy: 0.6815
Epoch 11: val_accuracy improved from 0.65887 to 0.66512, saving model to model2_rgb.h5
472/472 [==============================] - 81s 172ms/step - loss: 0.7630 - accuracy: 0.6815 - val_loss: 0.8163 - val_accuracy: 0.6651 - lr: 0.0010
Epoch 12/20
472/472 [==============================] - ETA: 0s - loss: 0.7426 - accuracy: 0.6910
Epoch 12: val_accuracy did not improve from 0.66512
472/472 [==============================] - 82s 174ms/step - loss: 0.7426 - accuracy: 0.6910 - val_loss: 0.8572 - val_accuracy: 0.6490 - lr: 0.0010
Epoch 13/20
472/472 [==============================] - ETA: 0s - loss: 0.7290 - accuracy: 0.6964
Epoch 13: val_accuracy improved from 0.66512 to 0.68185, saving model to model2_rgb.h5
472/472 [==============================] - 83s 176ms/step - loss: 0.7290 - accuracy: 0.6964 - val_loss: 0.7796 - val_accuracy: 0.6819 - lr: 0.0010
Epoch 14/20
472/472 [==============================] - ETA: 0s - loss: 0.7147 - accuracy: 0.7033
Epoch 14: val_accuracy did not improve from 0.68185
472/472 [==============================] - 84s 177ms/step - loss: 0.7147 - accuracy: 0.7033 - val_loss: 0.8151 - val_accuracy: 0.6629 - lr: 0.0010
Epoch 15/20
472/472 [==============================] - ETA: 0s - loss: 0.7002 - accuracy: 0.7074
Epoch 15: val_accuracy did not improve from 0.68185
472/472 [==============================] - 82s 175ms/step - loss: 0.7002 - accuracy: 0.7074 - val_loss: 0.8035 - val_accuracy: 0.6746 - lr: 0.0010
Epoch 16/20
472/472 [==============================] - ETA: 0s - loss: 0.6898 - accuracy: 0.7120Restoring model weights from the end of the best epoch: 13.

Epoch 16: val_accuracy did not improve from 0.68185

Epoch 16: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026.
472/472 [==============================] - 83s 175ms/step - loss: 0.6898 - accuracy: 0.7120 - val_loss: 0.7956 - val_accuracy: 0.6802 - lr: 0.0010
Epoch 16: early stopping

Evaluating the Model on the Test Set

# Load the saved model if notebook session gets disconnected
model2_rgb = load_model("model2_rgb.h5")

# Evaluate the model on the test data using RGB model
test_loss, test_accuracy = model2_rgb.evaluate(test_set_rgb, verbose=1)

# Print the results
print("Test Loss:", test_loss)
print("Test Accuracy:", test_accuracy)

4/4 [==============================] - 0s 45ms/step - loss: 0.7281 - accuracy: 0.6719
Test Loss: 0.728143572807312
Test Accuracy: 0.671875

# Plotting the Training and Validation accuracies of CNN model 2 and RGB images
plt.plot(history_model2_rgb.history['accuracy'])
plt.plot(history_model2_rgb.history['val_accuracy'])
plt.title('CNN Model 2 Accuracy with RGB Images')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc = 'upper left')

# Display the plot
plt.show()

Trying out Model 2 with Greyscale dataset

# Clearing backend
backend.clear_session()

# Reset seed for number generators using reset_seed() function
reset_seed(fixed_seed)

Random number generators seed has been set to 42

# ModelCheckpoint is used to save the model at certain intervals
checkpoint_model2_gray = ModelCheckpoint("model2_grayscale.h5",  # file path where the model will be saved
                             monitor='val_accuracy',          # means the callback monitors the validation accuracy
                             verbose=1,                       # model is saved only when the metric (val_acc) has improved
                             save_best_only=True,
                             mode='max')

# Configured callbacks will be passed to the model during training
callbacks_list_model2_gray = [early_stopping, checkpoint_model2_gray, reduce_learningrate]

# Number of iterations over the entire dataset that the training process should run
epochs = 20

# Building the model
model2_gray = cnn_model_2(1)
# Print first model's summary
model2_gray.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 256)       1280      
                                                                 
 batch_normalization (Batch  (None, 48, 48, 256)       1024      
 Normalization)                                                  
                                                                 
 leaky_re_lu (LeakyReLU)     (None, 48, 48, 256)       0         
                                                                 
 max_pooling2d (MaxPooling2  (None, 24, 24, 256)       0         
 D)                                                              
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 128)       131200    
                                                                 
 batch_normalization_1 (Bat  (None, 24, 24, 128)       512       
 chNormalization)                                                
                                                                 
 leaky_re_lu_1 (LeakyReLU)   (None, 24, 24, 128)       0         
                                                                 
 max_pooling2d_1 (MaxPoolin  (None, 12, 12, 128)       0         
 g2D)                                                            
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 64)        32832     
                                                                 
 batch_normalization_2 (Bat  (None, 12, 12, 64)        256       
 chNormalization)                                                
                                                                 
 leaky_re_lu_2 (LeakyReLU)   (None, 12, 12, 64)        0         
                                                                 
 max_pooling2d_2 (MaxPoolin  (None, 6, 6, 64)          0         
 g2D)                                                            
                                                                 
 conv2d_3 (Conv2D)           (None, 6, 6, 32)          8224      
                                                                 
 batch_normalization_3 (Bat  (None, 6, 6, 32)          128       
 chNormalization)                                                
                                                                 
 leaky_re_lu_3 (LeakyReLU)   (None, 6, 6, 32)          0         
                                                                 
 max_pooling2d_3 (MaxPoolin  (None, 3, 3, 32)          0         
 g2D)                                                            
                                                                 
 flatten (Flatten)           (None, 288)               0         
                                                                 
 dense (Dense)               (None, 512)               147968    
                                                                 
 dense_1 (Dense)             (None, 128)               65664     
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_2 (Dense)             (None, 4)                 516       
                                                                 
=================================================================
Total params: 389604 (1.49 MB)
Trainable params: 388644 (1.48 MB)
Non-trainable params: 960 (3.75 KB)
_________________________________________________________________

# Fitting the model with required parameters above, saving model into history_model2_gray variable
history_model2_gray = model2_gray.fit(
    train_set_gray,
    steps_per_epoch = train_set_gray.samples // train_set_gray.batch_size,
    validation_data = validation_set_gray,
    validation_steps = validation_set_gray.samples // validation_set_gray.batch_size,
    epochs = 20,
    verbose = 1,
    callbacks = callbacks_list_model2_gray
)

Epoch 1/20
472/472 [==============================] - ETA: 0s - loss: 1.3722 - accuracy: 0.3058
Epoch 1: val_accuracy improved from -inf to 0.39980, saving model to model2_grayscale.h5
472/472 [==============================] - 85s 180ms/step - loss: 1.3722 - accuracy: 0.3058 - val_loss: 1.3364 - val_accuracy: 0.3998 - lr: 0.0010
Epoch 2/20
472/472 [==============================] - ETA: 0s - loss: 1.2458 - accuracy: 0.3947
Epoch 2: val_accuracy improved from 0.39980 to 0.44214, saving model to model2_grayscale.h5
472/472 [==============================] - 86s 182ms/step - loss: 1.2458 - accuracy: 0.3947 - val_loss: 1.2179 - val_accuracy: 0.4421 - lr: 0.0010
Epoch 3/20
472/472 [==============================] - ETA: 0s - loss: 1.1441 - accuracy: 0.4674
Epoch 3: val_accuracy improved from 0.44214 to 0.44738, saving model to model2_grayscale.h5
472/472 [==============================] - 86s 182ms/step - loss: 1.1441 - accuracy: 0.4674 - val_loss: 1.1624 - val_accuracy: 0.4474 - lr: 0.0010
Epoch 4/20
472/472 [==============================] - ETA: 0s - loss: 1.0328 - accuracy: 0.5486
Epoch 4: val_accuracy improved from 0.44738 to 0.54536, saving model to model2_grayscale.h5
472/472 [==============================] - 86s 183ms/step - loss: 1.0328 - accuracy: 0.5486 - val_loss: 1.0824 - val_accuracy: 0.5454 - lr: 0.0010
Epoch 5/20
472/472 [==============================] - ETA: 0s - loss: 0.9544 - accuracy: 0.5885
Epoch 5: val_accuracy improved from 0.54536 to 0.60786, saving model to model2_grayscale.h5
472/472 [==============================] - 87s 185ms/step - loss: 0.9544 - accuracy: 0.5885 - val_loss: 0.9520 - val_accuracy: 0.6079 - lr: 0.0010
Epoch 6/20
472/472 [==============================] - ETA: 0s - loss: 0.9004 - accuracy: 0.6175
Epoch 6: val_accuracy improved from 0.60786 to 0.63710, saving model to model2_grayscale.h5
472/472 [==============================] - 85s 180ms/step - loss: 0.9004 - accuracy: 0.6175 - val_loss: 0.8746 - val_accuracy: 0.6371 - lr: 0.0010
Epoch 7/20
472/472 [==============================] - ETA: 0s - loss: 0.8636 - accuracy: 0.6322
Epoch 7: val_accuracy did not improve from 0.63710
472/472 [==============================] - 87s 183ms/step - loss: 0.8636 - accuracy: 0.6322 - val_loss: 0.9433 - val_accuracy: 0.6095 - lr: 0.0010
Epoch 8/20
472/472 [==============================] - ETA: 0s - loss: 0.8300 - accuracy: 0.6528
Epoch 8: val_accuracy improved from 0.63710 to 0.65464, saving model to model2_grayscale.h5
472/472 [==============================] - 86s 182ms/step - loss: 0.8300 - accuracy: 0.6528 - val_loss: 0.8286 - val_accuracy: 0.6546 - lr: 0.0010
Epoch 9/20
472/472 [==============================] - ETA: 0s - loss: 0.8099 - accuracy: 0.6596
Epoch 9: val_accuracy improved from 0.65464 to 0.65988, saving model to model2_grayscale.h5
472/472 [==============================] - 86s 183ms/step - loss: 0.8099 - accuracy: 0.6596 - val_loss: 0.8221 - val_accuracy: 0.6599 - lr: 0.0010
Epoch 10/20
472/472 [==============================] - ETA: 0s - loss: 0.7813 - accuracy: 0.6725
Epoch 10: val_accuracy did not improve from 0.65988
472/472 [==============================] - 84s 177ms/step - loss: 0.7813 - accuracy: 0.6725 - val_loss: 0.8526 - val_accuracy: 0.6470 - lr: 0.0010
Epoch 11/20
472/472 [==============================] - ETA: 0s - loss: 0.7628 - accuracy: 0.6816
Epoch 11: val_accuracy improved from 0.65988 to 0.66794, saving model to model2_grayscale.h5
472/472 [==============================] - 84s 178ms/step - loss: 0.7628 - accuracy: 0.6816 - val_loss: 0.8061 - val_accuracy: 0.6679 - lr: 0.0010
Epoch 12/20
472/472 [==============================] - ETA: 0s - loss: 0.7431 - accuracy: 0.6893
Epoch 12: val_accuracy did not improve from 0.66794
472/472 [==============================] - 84s 178ms/step - loss: 0.7431 - accuracy: 0.6893 - val_loss: 0.8002 - val_accuracy: 0.6677 - lr: 0.0010
Epoch 13/20
472/472 [==============================] - ETA: 0s - loss: 0.7252 - accuracy: 0.6987
Epoch 13: val_accuracy did not improve from 0.66794
472/472 [==============================] - 84s 179ms/step - loss: 0.7252 - accuracy: 0.6987 - val_loss: 0.8336 - val_accuracy: 0.6639 - lr: 0.0010
Epoch 14/20
472/472 [==============================] - ETA: 0s - loss: 0.7122 - accuracy: 0.7021
Epoch 14: val_accuracy did not improve from 0.66794
472/472 [==============================] - 83s 175ms/step - loss: 0.7122 - accuracy: 0.7021 - val_loss: 0.8503 - val_accuracy: 0.6456 - lr: 0.0010
Epoch 15/20
472/472 [==============================] - ETA: 0s - loss: 0.6940 - accuracy: 0.7109Restoring model weights from the end of the best epoch: 12.

Epoch 15: val_accuracy improved from 0.66794 to 0.67984, saving model to model2_grayscale.h5

Epoch 15: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026.
472/472 [==============================] - 84s 178ms/step - loss: 0.6940 - accuracy: 0.7109 - val_loss: 0.8075 - val_accuracy: 0.6798 - lr: 0.0010
Epoch 15: early stopping

# Load the saved model if notebook session gets disconnected
model2_gray = load_model("model2_grayscale.h5")

# Evaluate the model on the test data using Grayscale model
test_loss, test_accuracy = model2_gray.evaluate(test_set_gray, verbose=1)

# Print the results
print("Test Loss:", test_loss)
print("Test Accuracy:", test_accuracy)

4/4 [==============================] - 0s 40ms/step - loss: 0.7769 - accuracy: 0.6250
Test Loss: 0.7768638134002686
Test Accuracy: 0.625

# Plotting the Training and Validation accuracies of CNN model 2 and Grayscale images
plt.plot(history_model2_gray.history['accuracy'])
plt.plot(history_model2_gray.history['val_accuracy'])
plt.title('CNN Model 2 Accuracy with Grayscale Images')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc = 'upper left')

# Display the plot
plt.show()

Observations and Insights:

• The Greyscale model and RGB model have performed very similarly in both training and in validation.
  • RGB: 71.2% training accuracy, 68.2% validation accuracy.
  • Greyscale: 71.1% training accuracy, 68% validation accuracy.
• Considering that the training has stopped early at epoch 15, it might be worth considering relaxing the conditions for the early stopping in an effort of improving the accuracy of the network with further training.
• A slightly faster processing time per epoch would allow us to train the Grayscale model further, therefore I would select the Greyscale model as opposed to the RGB model.
• Not only the Greyscale model processing time is slightly faster, but also more suitable for its input dataset images which are in Greyscale originally.
• Again, the accuracy plot shows that the models haven't potentially platoed in terms of efficiency improvement over the epochs. Perhaps relaxing the conditions at which Tensorflow stops the training could have allowed the model to continue to improve over time.

The Greyscale models have performed very similarly to RGB models in both situations where the CNN was less or more complex. With the points above in mind, I would pick the more complex Greyscale model, considering that the processing time is slightly less when compared to RGB models and that the images in the dataset are originally in Greyscale.

Transfer Learning Architectures

In this section, we will create several Transfer Learning architectures. For the pre-trained models, we will select three popular architectures namely, VGG16, ResNet v2, and Efficient Net. The difference between these architectures and the previous architectures is that these will require 3 input channels while the earlier ones worked on 'grayscale' images. Therefore, we need to create new DataLoaders.

Creating our Data Loaders for Transfer Learning Architectures

In this section, we are creating data loaders that we will use as inputs to our Neural Network. We will have to go with color_mode = 'rgb' as this is the required format for the transfer learning architectures.

# Creating dataloader for train for RGB dataset
train_set_pre = datagen.flow_from_directory(folder_path + "train",
                                        target_size = (img_size),
                                        color_mode = 'rgb',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        classes = ['happy', 'sad', 'neutral', 'surprise'],
                                        shuffle = True)

# Creating dataloader for validation for RGB dataset
validation_set_pre = datagen.flow_from_directory(folder_path + "validation",
                                        target_size = (img_size),
                                        color_mode = 'rgb',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        classes = ['happy', 'sad', 'neutral', 'surprise'],
                                        shuffle = True)

# Creating dataloader for test for RGB dataset
test_set_pre = datagen.flow_from_directory(folder_path + "test",
                                        target_size = (img_size),
                                        color_mode = 'rgb',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        classes = ['happy', 'sad', 'neutral', 'surprise'],
                                        shuffle = True)

Found 15109 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.
Found 128 images belonging to 4 classes.

VGG16 Model

Importing the VGG16 Architecture

# Clearing backend
backend.clear_session()

# Reset seed for number generators using reset_seed() function
reset_seed(fixed_seed)

Random number generators seed has been set to 42

Model Building

• Import VGG16 upto the layer of your choice and add Fully Connected layers on top of it.

from keras.applications import VGG16
from keras.models import Model
from keras.layers import Flatten, Dense, Dropout, BatchNormalization, Input

# Load the VGG16 model without top layers
vgg_base = VGG16(include_top=False, 
                 weights='imagenet', 
                 input_tensor=Input(shape=(img_size[0],img_size[1], 3)))

# Set the VGG16 model to be non-trainable
vgg_base.trainable = False

# Rebuild the model
x = vgg_base.output
x = Flatten()(x)
x = Dense(256, activation='relu')(x)
x = Dense(128, activation='relu')(x)
x = Dropout(0.3)(x)
x = Dense(64, activation='relu')(x)
x = BatchNormalization()(x)
pred = Dense(4, activation='softmax')(x)

# Create a new model
vggmodel = Model(inputs=vgg_base.input, outputs=pred)

# Print the model summary
vggmodel.summary()

Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 48, 48, 3)]       0         
                                                                 
 block1_conv1 (Conv2D)       (None, 48, 48, 64)        1792      
                                                                 
 block1_conv2 (Conv2D)       (None, 48, 48, 64)        36928     
                                                                 
 block1_pool (MaxPooling2D)  (None, 24, 24, 64)        0         
                                                                 
 block2_conv1 (Conv2D)       (None, 24, 24, 128)       73856     
                                                                 
 block2_conv2 (Conv2D)       (None, 24, 24, 128)       147584    
                                                                 
 block2_pool (MaxPooling2D)  (None, 12, 12, 128)       0         
                                                                 
 block3_conv1 (Conv2D)       (None, 12, 12, 256)       295168    
                                                                 
 block3_conv2 (Conv2D)       (None, 12, 12, 256)       590080    
                                                                 
 block3_conv3 (Conv2D)       (None, 12, 12, 256)       590080    
                                                                 
 block3_pool (MaxPooling2D)  (None, 6, 6, 256)         0         
                                                                 
 block4_conv1 (Conv2D)       (None, 6, 6, 512)         1180160   
                                                                 
 block4_conv2 (Conv2D)       (None, 6, 6, 512)         2359808   
                                                                 
 block4_conv3 (Conv2D)       (None, 6, 6, 512)         2359808   
                                                                 
 block4_pool (MaxPooling2D)  (None, 3, 3, 512)         0         
                                                                 
 block5_conv1 (Conv2D)       (None, 3, 3, 512)         2359808   
                                                                 
 block5_conv2 (Conv2D)       (None, 3, 3, 512)         2359808   
                                                                 
 block5_conv3 (Conv2D)       (None, 3, 3, 512)         2359808   
                                                                 
 block5_pool (MaxPooling2D)  (None, 1, 1, 512)         0         
                                                                 
 flatten (Flatten)           (None, 512)               0         
                                                                 
 dense (Dense)               (None, 256)               131328    
                                                                 
 dense_1 (Dense)             (None, 128)               32896     
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_2 (Dense)             (None, 64)                8256      
                                                                 
 batch_normalization (Batch  (None, 64)                256       
 Normalization)                                                  
                                                                 
 dense_3 (Dense)             (None, 4)                 260       
                                                                 
=================================================================
Total params: 14887684 (56.79 MB)
Trainable params: 172868 (675.27 KB)
Non-trainable params: 14714816 (56.13 MB)
_________________________________________________________________

# Visualize the model as blocks
plot_model(vggmodel, to_file='model.png', show_shapes=True, show_layer_names=True)

Compiling and Training the VGG16 Model

# ModelCheckpoint is used to save the model at certain intervals
checkpoint_vgg = ModelCheckpoint("vggmodel.h5", # file path where the model will be saved
                             monitor='val_accuracy',  # means the callback monitors the validation accuracy
                             verbose=1,               # model is saved only when the metric (val_acc) has improved
                             save_best_only=True,
                             mode='max')

# Configured callbacks will be passed to the model during training
callbacks_list_vgg = [early_stopping, checkpoint_vgg, reduce_learningrate]

# Number of iterations over the entire dataset that the training process should run
epochs = 20

# Compiling the CNN model with categorical crossentropy as loss function, Adam Optimizer
# with 0.001 learning rate, and set metrics set to 'accuracy'.
vggmodel.compile(loss='categorical_crossentropy',
          optimizer=legacy.Adam(learning_rate = 0.001),
          metrics=['accuracy'])

# Fitting the model with required parameters above, saving model into history_vgg variable
history_vgg = vggmodel.fit(
    train_set_pre,
    steps_per_epoch = train_set_pre.samples // train_set_pre.batch_size,
    validation_data = validation_set_pre,
    validation_steps = validation_set_pre.samples // validation_set_pre.batch_size,
    epochs = 20,
    verbose = 1,
    callbacks = callbacks_list_vgg
)

Epoch 1/20
472/472 [==============================] - ETA: 0s - loss: 1.3230 - accuracy: 0.3808
Epoch 1: val_accuracy improved from -inf to 0.41250, saving model to vggmodel.h5
472/472 [==============================] - 110s 233ms/step - loss: 1.3230 - accuracy: 0.3808 - val_loss: 1.2388 - val_accuracy: 0.4125 - lr: 0.0010
Epoch 2/20

/Users/ceb/anaconda3/lib/python3.11/site-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

472/472 [==============================] - ETA: 0s - loss: 1.2147 - accuracy: 0.4441
Epoch 2: val_accuracy improved from 0.41250 to 0.48629, saving model to vggmodel.h5
472/472 [==============================] - 104s 219ms/step - loss: 1.2147 - accuracy: 0.4441 - val_loss: 1.1530 - val_accuracy: 0.4863 - lr: 0.0010
Epoch 3/20
472/472 [==============================] - ETA: 0s - loss: 1.1849 - accuracy: 0.4615
Epoch 3: val_accuracy improved from 0.48629 to 0.50121, saving model to vggmodel.h5
472/472 [==============================] - 95s 202ms/step - loss: 1.1849 - accuracy: 0.4615 - val_loss: 1.1402 - val_accuracy: 0.5012 - lr: 0.0010
Epoch 4/20
472/472 [==============================] - ETA: 0s - loss: 1.1698 - accuracy: 0.4722
Epoch 4: val_accuracy did not improve from 0.50121
472/472 [==============================] - 97s 206ms/step - loss: 1.1698 - accuracy: 0.4722 - val_loss: 1.1603 - val_accuracy: 0.4645 - lr: 0.0010
Epoch 5/20
472/472 [==============================] - ETA: 0s - loss: 1.1534 - accuracy: 0.4791
Epoch 5: val_accuracy did not improve from 0.50121
472/472 [==============================] - 97s 206ms/step - loss: 1.1534 - accuracy: 0.4791 - val_loss: 1.1310 - val_accuracy: 0.4968 - lr: 0.0010
Epoch 6/20
472/472 [==============================] - ETA: 0s - loss: 1.1497 - accuracy: 0.4814
Epoch 6: val_accuracy did not improve from 0.50121
472/472 [==============================] - 98s 206ms/step - loss: 1.1497 - accuracy: 0.4814 - val_loss: 1.1381 - val_accuracy: 0.4875 - lr: 0.0010
Epoch 7/20
472/472 [==============================] - ETA: 0s - loss: 1.1446 - accuracy: 0.4884
Epoch 7: val_accuracy improved from 0.50121 to 0.50464, saving model to vggmodel.h5
472/472 [==============================] - 97s 205ms/step - loss: 1.1446 - accuracy: 0.4884 - val_loss: 1.1194 - val_accuracy: 0.5046 - lr: 0.0010
Epoch 8/20
472/472 [==============================] - ETA: 0s - loss: 1.1359 - accuracy: 0.4970
Epoch 8: val_accuracy did not improve from 0.50464
472/472 [==============================] - 99s 211ms/step - loss: 1.1359 - accuracy: 0.4970 - val_loss: 1.1455 - val_accuracy: 0.4804 - lr: 0.0010
Epoch 9/20
472/472 [==============================] - ETA: 0s - loss: 1.1270 - accuracy: 0.5039
Epoch 9: val_accuracy did not improve from 0.50464
472/472 [==============================] - 96s 204ms/step - loss: 1.1270 - accuracy: 0.5039 - val_loss: 1.1971 - val_accuracy: 0.4681 - lr: 0.0010
Epoch 10/20
472/472 [==============================] - ETA: 0s - loss: 1.1223 - accuracy: 0.5020Restoring model weights from the end of the best epoch: 7.

Epoch 10: val_accuracy did not improve from 0.50464

Epoch 10: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026.
472/472 [==============================] - 99s 210ms/step - loss: 1.1223 - accuracy: 0.5020 - val_loss: 1.1743 - val_accuracy: 0.4744 - lr: 0.0010
Epoch 10: early stopping

Evaluating the VGG16 model

# Load the saved model if notebook session gets disconnected
vggmodel = load_model("vggmodel.h5")

# Evaluate the model on the test data
test_loss, test_accuracy = vggmodel.evaluate(test_set_pre, verbose=1)

# Print the results
print("Test Loss:", test_loss)
print("Test Accuracy:", test_accuracy)

4/4 [==============================] - 1s 158ms/step - loss: 1.2255 - accuracy: 0.4453
Test Loss: 1.225456714630127
Test Accuracy: 0.4453125

# Plotting the Training and Validation accuracies of VGG16
plt.plot(history_vgg.history['accuracy'])
plt.plot(history_vgg.history['val_accuracy'])
plt.title('VGG16 Transfer Learning Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc = 'upper left')

# Display the plot
plt.show()

Observations and Insights:

• Although training time has very significantly increased, the VGG16 model has performed significantly worse in both training and validation than our custom CNN model for Greyscale images previously chosen.
  • Chosen CNN for Greyscale images: 71.1% training accuracy, 68% validation accuracy.
  • VGG16: 50.2% training accuracy, 47.4% validation accuracy.
• The decrease in validation accuracy in later epochs could be indicative of overfitting, where the model becomes too tailored to the training data and loses its predictive power on new, unseen data.
• Training has stopped very prematurely due to Learning Rate platoeing. Perhaps relaxing the conditions at which the condition is met might allow the model to improve its performance over time or could lead to further overfitting.

The VGG16 model hasn't performed better than our selected CNN Model for Greyscale images. An early stopping was triggered at epoch 10, which might have been due to the model not generalising enough for the task. Considering the above, the CNN Model for Greyscale images is still the best performing model thus far.

ResNet V2 Model

# Clearing backend
backend.clear_session()

# Reset seed for number generators using reset_seed() function
reset_seed(fixed_seed)

Random number generators seed has been set to 42

import tensorflow as tf
from tensorflow.keras.applications import ResNet50
from tensorflow.keras.layers import Flatten, Dense, GlobalAveragePooling2D

# Load the ResNet50 model without its top layers and with pre-trained ImageNet weights
Resnet = ResNet50(include_top=False, weights='imagenet', input_tensor=tf.keras.Input(shape=(img_size[0], img_size[1], 3)))

# Print imported model's summary
Resnet.summary()

Model: "resnet50"
__________________________________________________________________________________________________
 Layer (type)                Output Shape                 Param #   Connected to                  
==================================================================================================
 input_1 (InputLayer)        [(None, 48, 48, 3)]          0         []                            
                                                                                                  
 conv1_pad (ZeroPadding2D)   (None, 54, 54, 3)            0         ['input_1[0][0]']             
                                                                                                  
 conv1_conv (Conv2D)         (None, 24, 24, 64)           9472      ['conv1_pad[0][0]']           
                                                                                                  
 conv1_bn (BatchNormalizati  (None, 24, 24, 64)           256       ['conv1_conv[0][0]']          
 on)                                                                                              
                                                                                                  
 conv1_relu (Activation)     (None, 24, 24, 64)           0         ['conv1_bn[0][0]']            
                                                                                                  
 pool1_pad (ZeroPadding2D)   (None, 26, 26, 64)           0         ['conv1_relu[0][0]']          
                                                                                                  
 pool1_pool (MaxPooling2D)   (None, 12, 12, 64)           0         ['pool1_pad[0][0]']           
                                                                                                  
 conv2_block1_1_conv (Conv2  (None, 12, 12, 64)           4160      ['pool1_pool[0][0]']          
 D)                                                                                               
                                                                                                  
 conv2_block1_1_bn (BatchNo  (None, 12, 12, 64)           256       ['conv2_block1_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block1_1_relu (Activ  (None, 12, 12, 64)           0         ['conv2_block1_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv2_block1_2_conv (Conv2  (None, 12, 12, 64)           36928     ['conv2_block1_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv2_block1_2_bn (BatchNo  (None, 12, 12, 64)           256       ['conv2_block1_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block1_2_relu (Activ  (None, 12, 12, 64)           0         ['conv2_block1_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv2_block1_0_conv (Conv2  (None, 12, 12, 256)          16640     ['pool1_pool[0][0]']          
 D)                                                                                               
                                                                                                  
 conv2_block1_3_conv (Conv2  (None, 12, 12, 256)          16640     ['conv2_block1_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv2_block1_0_bn (BatchNo  (None, 12, 12, 256)          1024      ['conv2_block1_0_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block1_3_bn (BatchNo  (None, 12, 12, 256)          1024      ['conv2_block1_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block1_add (Add)      (None, 12, 12, 256)          0         ['conv2_block1_0_bn[0][0]',   
                                                                     'conv2_block1_3_bn[0][0]']   
                                                                                                  
 conv2_block1_out (Activati  (None, 12, 12, 256)          0         ['conv2_block1_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv2_block2_1_conv (Conv2  (None, 12, 12, 64)           16448     ['conv2_block1_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv2_block2_1_bn (BatchNo  (None, 12, 12, 64)           256       ['conv2_block2_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block2_1_relu (Activ  (None, 12, 12, 64)           0         ['conv2_block2_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv2_block2_2_conv (Conv2  (None, 12, 12, 64)           36928     ['conv2_block2_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv2_block2_2_bn (BatchNo  (None, 12, 12, 64)           256       ['conv2_block2_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block2_2_relu (Activ  (None, 12, 12, 64)           0         ['conv2_block2_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv2_block2_3_conv (Conv2  (None, 12, 12, 256)          16640     ['conv2_block2_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv2_block2_3_bn (BatchNo  (None, 12, 12, 256)          1024      ['conv2_block2_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block2_add (Add)      (None, 12, 12, 256)          0         ['conv2_block1_out[0][0]',    
                                                                     'conv2_block2_3_bn[0][0]']   
                                                                                                  
 conv2_block2_out (Activati  (None, 12, 12, 256)          0         ['conv2_block2_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv2_block3_1_conv (Conv2  (None, 12, 12, 64)           16448     ['conv2_block2_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv2_block3_1_bn (BatchNo  (None, 12, 12, 64)           256       ['conv2_block3_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block3_1_relu (Activ  (None, 12, 12, 64)           0         ['conv2_block3_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv2_block3_2_conv (Conv2  (None, 12, 12, 64)           36928     ['conv2_block3_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv2_block3_2_bn (BatchNo  (None, 12, 12, 64)           256       ['conv2_block3_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block3_2_relu (Activ  (None, 12, 12, 64)           0         ['conv2_block3_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv2_block3_3_conv (Conv2  (None, 12, 12, 256)          16640     ['conv2_block3_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv2_block3_3_bn (BatchNo  (None, 12, 12, 256)          1024      ['conv2_block3_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block3_add (Add)      (None, 12, 12, 256)          0         ['conv2_block2_out[0][0]',    
                                                                     'conv2_block3_3_bn[0][0]']   
                                                                                                  
 conv2_block3_out (Activati  (None, 12, 12, 256)          0         ['conv2_block3_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv3_block1_1_conv (Conv2  (None, 6, 6, 128)            32896     ['conv2_block3_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv3_block1_1_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block1_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block1_1_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block1_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block1_2_conv (Conv2  (None, 6, 6, 128)            147584    ['conv3_block1_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block1_2_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block1_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block1_2_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block1_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block1_0_conv (Conv2  (None, 6, 6, 512)            131584    ['conv2_block3_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv3_block1_3_conv (Conv2  (None, 6, 6, 512)            66048     ['conv3_block1_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block1_0_bn (BatchNo  (None, 6, 6, 512)            2048      ['conv3_block1_0_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block1_3_bn (BatchNo  (None, 6, 6, 512)            2048      ['conv3_block1_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block1_add (Add)      (None, 6, 6, 512)            0         ['conv3_block1_0_bn[0][0]',   
                                                                     'conv3_block1_3_bn[0][0]']   
                                                                                                  
 conv3_block1_out (Activati  (None, 6, 6, 512)            0         ['conv3_block1_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv3_block2_1_conv (Conv2  (None, 6, 6, 128)            65664     ['conv3_block1_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv3_block2_1_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block2_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block2_1_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block2_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block2_2_conv (Conv2  (None, 6, 6, 128)            147584    ['conv3_block2_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block2_2_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block2_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block2_2_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block2_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block2_3_conv (Conv2  (None, 6, 6, 512)            66048     ['conv3_block2_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block2_3_bn (BatchNo  (None, 6, 6, 512)            2048      ['conv3_block2_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block2_add (Add)      (None, 6, 6, 512)            0         ['conv3_block1_out[0][0]',    
                                                                     'conv3_block2_3_bn[0][0]']   
                                                                                                  
 conv3_block2_out (Activati  (None, 6, 6, 512)            0         ['conv3_block2_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv3_block3_1_conv (Conv2  (None, 6, 6, 128)            65664     ['conv3_block2_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv3_block3_1_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block3_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block3_1_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block3_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block3_2_conv (Conv2  (None, 6, 6, 128)            147584    ['conv3_block3_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block3_2_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block3_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block3_2_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block3_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block3_3_conv (Conv2  (None, 6, 6, 512)            66048     ['conv3_block3_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block3_3_bn (BatchNo  (None, 6, 6, 512)            2048      ['conv3_block3_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block3_add (Add)      (None, 6, 6, 512)            0         ['conv3_block2_out[0][0]',    
                                                                     'conv3_block3_3_bn[0][0]']   
                                                                                                  
 conv3_block3_out (Activati  (None, 6, 6, 512)            0         ['conv3_block3_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv3_block4_1_conv (Conv2  (None, 6, 6, 128)            65664     ['conv3_block3_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv3_block4_1_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block4_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block4_1_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block4_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block4_2_conv (Conv2  (None, 6, 6, 128)            147584    ['conv3_block4_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block4_2_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block4_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block4_2_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block4_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block4_3_conv (Conv2  (None, 6, 6, 512)            66048     ['conv3_block4_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block4_3_bn (BatchNo  (None, 6, 6, 512)            2048      ['conv3_block4_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block4_add (Add)      (None, 6, 6, 512)            0         ['conv3_block3_out[0][0]',    
                                                                     'conv3_block4_3_bn[0][0]']   
                                                                                                  
 conv3_block4_out (Activati  (None, 6, 6, 512)            0         ['conv3_block4_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block1_1_conv (Conv2  (None, 3, 3, 256)            131328    ['conv3_block4_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block1_1_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block1_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block1_1_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block1_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block1_2_conv (Conv2  (None, 3, 3, 256)            590080    ['conv4_block1_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block1_2_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block1_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block1_2_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block1_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block1_0_conv (Conv2  (None, 3, 3, 1024)           525312    ['conv3_block4_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block1_3_conv (Conv2  (None, 3, 3, 1024)           263168    ['conv4_block1_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block1_0_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block1_0_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block1_3_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block1_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block1_add (Add)      (None, 3, 3, 1024)           0         ['conv4_block1_0_bn[0][0]',   
                                                                     'conv4_block1_3_bn[0][0]']   
                                                                                                  
 conv4_block1_out (Activati  (None, 3, 3, 1024)           0         ['conv4_block1_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block2_1_conv (Conv2  (None, 3, 3, 256)            262400    ['conv4_block1_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block2_1_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block2_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block2_1_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block2_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block2_2_conv (Conv2  (None, 3, 3, 256)            590080    ['conv4_block2_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block2_2_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block2_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block2_2_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block2_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block2_3_conv (Conv2  (None, 3, 3, 1024)           263168    ['conv4_block2_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block2_3_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block2_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block2_add (Add)      (None, 3, 3, 1024)           0         ['conv4_block1_out[0][0]',    
                                                                     'conv4_block2_3_bn[0][0]']   
                                                                                                  
 conv4_block2_out (Activati  (None, 3, 3, 1024)           0         ['conv4_block2_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block3_1_conv (Conv2  (None, 3, 3, 256)            262400    ['conv4_block2_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block3_1_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block3_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block3_1_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block3_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block3_2_conv (Conv2  (None, 3, 3, 256)            590080    ['conv4_block3_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block3_2_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block3_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block3_2_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block3_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block3_3_conv (Conv2  (None, 3, 3, 1024)           263168    ['conv4_block3_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block3_3_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block3_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block3_add (Add)      (None, 3, 3, 1024)           0         ['conv4_block2_out[0][0]',    
                                                                     'conv4_block3_3_bn[0][0]']   
                                                                                                  
 conv4_block3_out (Activati  (None, 3, 3, 1024)           0         ['conv4_block3_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block4_1_conv (Conv2  (None, 3, 3, 256)            262400    ['conv4_block3_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block4_1_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block4_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block4_1_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block4_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block4_2_conv (Conv2  (None, 3, 3, 256)            590080    ['conv4_block4_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block4_2_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block4_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block4_2_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block4_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block4_3_conv (Conv2  (None, 3, 3, 1024)           263168    ['conv4_block4_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block4_3_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block4_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block4_add (Add)      (None, 3, 3, 1024)           0         ['conv4_block3_out[0][0]',    
                                                                     'conv4_block4_3_bn[0][0]']   
                                                                                                  
 conv4_block4_out (Activati  (None, 3, 3, 1024)           0         ['conv4_block4_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block5_1_conv (Conv2  (None, 3, 3, 256)            262400    ['conv4_block4_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block5_1_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block5_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block5_1_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block5_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block5_2_conv (Conv2  (None, 3, 3, 256)            590080    ['conv4_block5_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block5_2_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block5_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block5_2_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block5_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block5_3_conv (Conv2  (None, 3, 3, 1024)           263168    ['conv4_block5_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block5_3_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block5_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block5_add (Add)      (None, 3, 3, 1024)           0         ['conv4_block4_out[0][0]',    
                                                                     'conv4_block5_3_bn[0][0]']   
                                                                                                  
 conv4_block5_out (Activati  (None, 3, 3, 1024)           0         ['conv4_block5_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block6_1_conv (Conv2  (None, 3, 3, 256)            262400    ['conv4_block5_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block6_1_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block6_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block6_1_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block6_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block6_2_conv (Conv2  (None, 3, 3, 256)            590080    ['conv4_block6_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block6_2_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block6_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block6_2_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block6_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block6_3_conv (Conv2  (None, 3, 3, 1024)           263168    ['conv4_block6_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block6_3_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block6_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block6_add (Add)      (None, 3, 3, 1024)           0         ['conv4_block5_out[0][0]',    
                                                                     'conv4_block6_3_bn[0][0]']   
                                                                                                  
 conv4_block6_out (Activati  (None, 3, 3, 1024)           0         ['conv4_block6_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv5_block1_1_conv (Conv2  (None, 2, 2, 512)            524800    ['conv4_block6_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv5_block1_1_bn (BatchNo  (None, 2, 2, 512)            2048      ['conv5_block1_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block1_1_relu (Activ  (None, 2, 2, 512)            0         ['conv5_block1_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv5_block1_2_conv (Conv2  (None, 2, 2, 512)            2359808   ['conv5_block1_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv5_block1_2_bn (BatchNo  (None, 2, 2, 512)            2048      ['conv5_block1_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block1_2_relu (Activ  (None, 2, 2, 512)            0         ['conv5_block1_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv5_block1_0_conv (Conv2  (None, 2, 2, 2048)           2099200   ['conv4_block6_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv5_block1_3_conv (Conv2  (None, 2, 2, 2048)           1050624   ['conv5_block1_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv5_block1_0_bn (BatchNo  (None, 2, 2, 2048)           8192      ['conv5_block1_0_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block1_3_bn (BatchNo  (None, 2, 2, 2048)           8192      ['conv5_block1_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block1_add (Add)      (None, 2, 2, 2048)           0         ['conv5_block1_0_bn[0][0]',   
                                                                     'conv5_block1_3_bn[0][0]']   
                                                                                                  
 conv5_block1_out (Activati  (None, 2, 2, 2048)           0         ['conv5_block1_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv5_block2_1_conv (Conv2  (None, 2, 2, 512)            1049088   ['conv5_block1_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv5_block2_1_bn (BatchNo  (None, 2, 2, 512)            2048      ['conv5_block2_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block2_1_relu (Activ  (None, 2, 2, 512)            0         ['conv5_block2_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv5_block2_2_conv (Conv2  (None, 2, 2, 512)            2359808   ['conv5_block2_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv5_block2_2_bn (BatchNo  (None, 2, 2, 512)            2048      ['conv5_block2_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block2_2_relu (Activ  (None, 2, 2, 512)            0         ['conv5_block2_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv5_block2_3_conv (Conv2  (None, 2, 2, 2048)           1050624   ['conv5_block2_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv5_block2_3_bn (BatchNo  (None, 2, 2, 2048)           8192      ['conv5_block2_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block2_add (Add)      (None, 2, 2, 2048)           0         ['conv5_block1_out[0][0]',    
                                                                     'conv5_block2_3_bn[0][0]']   
                                                                                                  
 conv5_block2_out (Activati  (None, 2, 2, 2048)           0         ['conv5_block2_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv5_block3_1_conv (Conv2  (None, 2, 2, 512)            1049088   ['conv5_block2_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv5_block3_1_bn (BatchNo  (None, 2, 2, 512)            2048      ['conv5_block3_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block3_1_relu (Activ  (None, 2, 2, 512)            0         ['conv5_block3_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv5_block3_2_conv (Conv2  (None, 2, 2, 512)            2359808   ['conv5_block3_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv5_block3_2_bn (BatchNo  (None, 2, 2, 512)            2048      ['conv5_block3_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block3_2_relu (Activ  (None, 2, 2, 512)            0         ['conv5_block3_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv5_block3_3_conv (Conv2  (None, 2, 2, 2048)           1050624   ['conv5_block3_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv5_block3_3_bn (BatchNo  (None, 2, 2, 2048)           8192      ['conv5_block3_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block3_add (Add)      (None, 2, 2, 2048)           0         ['conv5_block2_out[0][0]',    
                                                                     'conv5_block3_3_bn[0][0]']   
                                                                                                  
 conv5_block3_out (Activati  (None, 2, 2, 2048)           0         ['conv5_block3_add[0][0]']    
 on)                                                                                              
                                                                                                  
==================================================================================================
Total params: 23587712 (89.98 MB)
Trainable params: 23534592 (89.78 MB)
Non-trainable params: 53120 (207.50 KB)
__________________________________________________________________________________________________

# Visualize the model as blocks
plot_model(Resnet, to_file='model.png', show_shapes=True, show_layer_names=True)

Model Building

# Set ResNet layers to non-trainable
Resnet.trainable = False

# Rebuild the model
x = Resnet.output
# Add Global Average Pooling layer
x = GlobalAveragePooling2D()(x)

# Flatten the output
x = Flatten()(x)

# Add a Dense layer with 256 neurons
x = Dense(256, activation='relu')(x)

# Add a Dense Layer with 128 neurons
x = Dense(128, activation='relu')(x)

# Add a DropOut layer with a drop out ratio of 0.3
x = Dropout(0.3)(x)

# Add a Dense Layer with 64 neurons
x = Dense(64, activation='relu')(x)

# Add a Batch Normalization layer
x = BatchNormalization()(x)

# Add the final dense layer with 4 neurons and 'softmax' activation for multi-class classification
pred = Dense(4, activation='softmax')(x)

# Initializing the model
resnetmodel = Model(inputs=Resnet.input, outputs=pred)

Compiling and Training the Model

# ModelCheckpoint is used to save the model at certain intervals
checkpoint_resnet = ModelCheckpoint("resnet.h5", # file path where the model will be saved
                    monitor='val_accuracy',  # means the callback monitors the validation accuracy
                    verbose=1,               # model is saved only when the metric (val_acc) has improved
                    save_best_only=True,
                    mode='max')

# Configured callbacks will be passed to the model during training
callbacks_list_resnet = [early_stopping, checkpoint_resnet, reduce_learningrate]

# Number of iterations over the entire dataset that the training process should run
epochs = 20

# Compiling the CNN model with categorical crossentropy as loss function, Adam Optimizer
# with 0.001 learning rate, and set metrics set to 'accuracy'.
resnetmodel.compile(loss='categorical_crossentropy',
          optimizer=legacy.Adam(learning_rate = 0.001),
          metrics=['accuracy'])

# Fitting the model with required parameters above, saving model into history_resnet variable
history_resnet = resnetmodel.fit(
    train_set_pre,
    steps_per_epoch = train_set_pre.samples // train_set_pre.batch_size,
    validation_data = validation_set_pre,
    validation_steps = validation_set_pre.samples // validation_set_pre.batch_size,
    epochs = 20,
    verbose = 1,
    callbacks = callbacks_list_resnet
)

Epoch 1/20
472/472 [==============================] - ETA: 0s - loss: 1.4481 - accuracy: 0.2605
Epoch 1: val_accuracy improved from -inf to 0.36996, saving model to resnet.h5
472/472 [==============================] - 47s 96ms/step - loss: 1.4481 - accuracy: 0.2605 - val_loss: 1.3482 - val_accuracy: 0.3700 - lr: 0.0010
Epoch 2/20
472/472 [==============================] - ETA: 0s - loss: 1.3908 - accuracy: 0.2780
Epoch 2: val_accuracy did not improve from 0.36996
472/472 [==============================] - 47s 99ms/step - loss: 1.3908 - accuracy: 0.2780 - val_loss: 1.3547 - val_accuracy: 0.2639 - lr: 0.0010
Epoch 3/20
472/472 [==============================] - ETA: 0s - loss: 1.3560 - accuracy: 0.3034
Epoch 3: val_accuracy did not improve from 0.36996
472/472 [==============================] - 51s 107ms/step - loss: 1.3560 - accuracy: 0.3034 - val_loss: 1.3632 - val_accuracy: 0.2653 - lr: 0.0010
Epoch 4/20
472/472 [==============================] - ETA: 0s - loss: 1.3313 - accuracy: 0.3308
Epoch 4: val_accuracy did not improve from 0.36996
472/472 [==============================] - 45s 95ms/step - loss: 1.3313 - accuracy: 0.3308 - val_loss: 1.3288 - val_accuracy: 0.3010 - lr: 0.0010
Epoch 5/20
472/472 [==============================] - ETA: 0s - loss: 1.3136 - accuracy: 0.3434
Epoch 5: val_accuracy did not improve from 0.36996
472/472 [==============================] - 46s 97ms/step - loss: 1.3136 - accuracy: 0.3434 - val_loss: 1.3745 - val_accuracy: 0.2597 - lr: 0.0010
Epoch 6/20
472/472 [==============================] - ETA: 0s - loss: 1.2930 - accuracy: 0.3601
Epoch 6: val_accuracy did not improve from 0.36996
472/472 [==============================] - 45s 94ms/step - loss: 1.2930 - accuracy: 0.3601 - val_loss: 1.3816 - val_accuracy: 0.2909 - lr: 0.0010
Epoch 7/20
472/472 [==============================] - ETA: 0s - loss: 1.2777 - accuracy: 0.3820Restoring model weights from the end of the best epoch: 4.

Epoch 7: val_accuracy did not improve from 0.36996

Epoch 7: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026.
472/472 [==============================] - 46s 97ms/step - loss: 1.2777 - accuracy: 0.3820 - val_loss: 1.3505 - val_accuracy: 0.3075 - lr: 0.0010
Epoch 7: early stopping

Evaluating the ResNet Model

# Load the saved model if notebook session gets disconnected
resnetmodel = load_model("resnet.h5")

# Evaluate the model on the test data
test_loss, test_accuracy = resnetmodel.evaluate(test_set_pre, verbose=1)

# Print the results
print("Test Loss:", test_loss)
print("Test Accuracy:", test_accuracy)

4/4 [==============================] - 0s 80ms/step - loss: 1.3008 - accuracy: 0.3750
Test Loss: 1.300801396369934
Test Accuracy: 0.375

# Plotting the Training and Validation accuracies of ResNet V2
plt.plot(history_resnet.history['accuracy'])
plt.plot(history_resnet.history['val_accuracy'])
plt.title('ResNet V2 Transfer Learning Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc = 'upper left')

# Display the plot
plt.show()

Observations and Insights:

• ResnetV2 model has performed significantly worse in both training and validation than our custom CNN model for Greyscale images previously chosen.
  • Chosen CNN for Greyscale images: 71.1% training accuracy, 68% validation accuracy.
  • VGG16: 38.2% training accuracy, 30.75% validation accuracy.
• Training has stopped very prematurely due to Learning Rate platoeing, which might indicate early overfitting from a model that does not generalises enough for this task. Perhaps relaxing the conditions at which the condition is met might allow the model to improve its performance over time or could lead to further overfitting.

ResnetV2 model hasn't performed better than our selected CNN Model for Greyscale images. An early stopping was triggered at a very early epoch 7, which might have been due to the model not generalising enough for the task. Considering the above, the CNN Model for Greyscale images is still the best performing model thus far.

EfficientNet Model

# Clearing backend
backend.clear_session()

# Reset seed for number generators using reset_seed() function
reset_seed(fixed_seed)

Random number generators seed has been set to 42

# Importing EfficientNet Model libraries
import tensorflow as tf
import tensorflow.keras.applications as ap
from tensorflow.keras import Model

# Variable 'EfficientNet' will contain the EfficientNet model
EfficientNet = ap.EfficientNetV2B2(include_top=False,
                                   weights="imagenet", 
                                   input_shape= (img_size[0], img_size[1], 3))
# Print imported model's summary
EfficientNet.summary()

Model: "efficientnetv2-b2"
__________________________________________________________________________________________________
 Layer (type)                Output Shape                 Param #   Connected to                  
==================================================================================================
 input_1 (InputLayer)        [(None, 48, 48, 3)]          0         []                            
                                                                                                  
 rescaling (Rescaling)       (None, 48, 48, 3)            0         ['input_1[0][0]']             
                                                                                                  
 normalization (Normalizati  (None, 48, 48, 3)            0         ['rescaling[0][0]']           
 on)                                                                                              
                                                                                                  
 stem_conv (Conv2D)          (None, 24, 24, 32)           864       ['normalization[0][0]']       
                                                                                                  
 stem_bn (BatchNormalizatio  (None, 24, 24, 32)           128       ['stem_conv[0][0]']           
 n)                                                                                               
                                                                                                  
 stem_activation (Activatio  (None, 24, 24, 32)           0         ['stem_bn[0][0]']             
 n)                                                                                               
                                                                                                  
 block1a_project_conv (Conv  (None, 24, 24, 16)           4608      ['stem_activation[0][0]']     
 2D)                                                                                              
                                                                                                  
 block1a_project_bn (BatchN  (None, 24, 24, 16)           64        ['block1a_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block1a_project_activation  (None, 24, 24, 16)           0         ['block1a_project_bn[0][0]']  
  (Activation)                                                                                    
                                                                                                  
 block1b_project_conv (Conv  (None, 24, 24, 16)           2304      ['block1a_project_activation[0
 2D)                                                                ][0]']                        
                                                                                                  
 block1b_project_bn (BatchN  (None, 24, 24, 16)           64        ['block1b_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block1b_project_activation  (None, 24, 24, 16)           0         ['block1b_project_bn[0][0]']  
  (Activation)                                                                                    
                                                                                                  
 block1b_drop (Dropout)      (None, 24, 24, 16)           0         ['block1b_project_activation[0
                                                                    ][0]']                        
                                                                                                  
 block1b_add (Add)           (None, 24, 24, 16)           0         ['block1b_drop[0][0]',        
                                                                     'block1a_project_activation[0
                                                                    ][0]']                        
                                                                                                  
 block2a_expand_conv (Conv2  (None, 12, 12, 64)           9216      ['block1b_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block2a_expand_bn (BatchNo  (None, 12, 12, 64)           256       ['block2a_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block2a_expand_activation   (None, 12, 12, 64)           0         ['block2a_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block2a_project_conv (Conv  (None, 12, 12, 32)           2048      ['block2a_expand_activation[0]
 2D)                                                                [0]']                         
                                                                                                  
 block2a_project_bn (BatchN  (None, 12, 12, 32)           128       ['block2a_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block2b_expand_conv (Conv2  (None, 12, 12, 128)          36864     ['block2a_project_bn[0][0]']  
 D)                                                                                               
                                                                                                  
 block2b_expand_bn (BatchNo  (None, 12, 12, 128)          512       ['block2b_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block2b_expand_activation   (None, 12, 12, 128)          0         ['block2b_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block2b_project_conv (Conv  (None, 12, 12, 32)           4096      ['block2b_expand_activation[0]
 2D)                                                                [0]']                         
                                                                                                  
 block2b_project_bn (BatchN  (None, 12, 12, 32)           128       ['block2b_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block2b_drop (Dropout)      (None, 12, 12, 32)           0         ['block2b_project_bn[0][0]']  
                                                                                                  
 block2b_add (Add)           (None, 12, 12, 32)           0         ['block2b_drop[0][0]',        
                                                                     'block2a_project_bn[0][0]']  
                                                                                                  
 block2c_expand_conv (Conv2  (None, 12, 12, 128)          36864     ['block2b_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block2c_expand_bn (BatchNo  (None, 12, 12, 128)          512       ['block2c_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block2c_expand_activation   (None, 12, 12, 128)          0         ['block2c_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block2c_project_conv (Conv  (None, 12, 12, 32)           4096      ['block2c_expand_activation[0]
 2D)                                                                [0]']                         
                                                                                                  
 block2c_project_bn (BatchN  (None, 12, 12, 32)           128       ['block2c_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block2c_drop (Dropout)      (None, 12, 12, 32)           0         ['block2c_project_bn[0][0]']  
                                                                                                  
 block2c_add (Add)           (None, 12, 12, 32)           0         ['block2c_drop[0][0]',        
                                                                     'block2b_add[0][0]']         
                                                                                                  
 block3a_expand_conv (Conv2  (None, 6, 6, 128)            36864     ['block2c_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block3a_expand_bn (BatchNo  (None, 6, 6, 128)            512       ['block3a_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block3a_expand_activation   (None, 6, 6, 128)            0         ['block3a_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block3a_project_conv (Conv  (None, 6, 6, 56)             7168      ['block3a_expand_activation[0]
 2D)                                                                [0]']                         
                                                                                                  
 block3a_project_bn (BatchN  (None, 6, 6, 56)             224       ['block3a_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block3b_expand_conv (Conv2  (None, 6, 6, 224)            112896    ['block3a_project_bn[0][0]']  
 D)                                                                                               
                                                                                                  
 block3b_expand_bn (BatchNo  (None, 6, 6, 224)            896       ['block3b_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block3b_expand_activation   (None, 6, 6, 224)            0         ['block3b_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block3b_project_conv (Conv  (None, 6, 6, 56)             12544     ['block3b_expand_activation[0]
 2D)                                                                [0]']                         
                                                                                                  
 block3b_project_bn (BatchN  (None, 6, 6, 56)             224       ['block3b_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block3b_drop (Dropout)      (None, 6, 6, 56)             0         ['block3b_project_bn[0][0]']  
                                                                                                  
 block3b_add (Add)           (None, 6, 6, 56)             0         ['block3b_drop[0][0]',        
                                                                     'block3a_project_bn[0][0]']  
                                                                                                  
 block3c_expand_conv (Conv2  (None, 6, 6, 224)            112896    ['block3b_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block3c_expand_bn (BatchNo  (None, 6, 6, 224)            896       ['block3c_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block3c_expand_activation   (None, 6, 6, 224)            0         ['block3c_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block3c_project_conv (Conv  (None, 6, 6, 56)             12544     ['block3c_expand_activation[0]
 2D)                                                                [0]']                         
                                                                                                  
 block3c_project_bn (BatchN  (None, 6, 6, 56)             224       ['block3c_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block3c_drop (Dropout)      (None, 6, 6, 56)             0         ['block3c_project_bn[0][0]']  
                                                                                                  
 block3c_add (Add)           (None, 6, 6, 56)             0         ['block3c_drop[0][0]',        
                                                                     'block3b_add[0][0]']         
                                                                                                  
 block4a_expand_conv (Conv2  (None, 6, 6, 224)            12544     ['block3c_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block4a_expand_bn (BatchNo  (None, 6, 6, 224)            896       ['block4a_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block4a_expand_activation   (None, 6, 6, 224)            0         ['block4a_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block4a_dwconv2 (Depthwise  (None, 3, 3, 224)            2016      ['block4a_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block4a_bn (BatchNormaliza  (None, 3, 3, 224)            896       ['block4a_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block4a_activation (Activa  (None, 3, 3, 224)            0         ['block4a_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block4a_se_squeeze (Global  (None, 224)                  0         ['block4a_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block4a_se_reshape (Reshap  (None, 1, 1, 224)            0         ['block4a_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block4a_se_reduce (Conv2D)  (None, 1, 1, 14)             3150      ['block4a_se_reshape[0][0]']  
                                                                                                  
 block4a_se_expand (Conv2D)  (None, 1, 1, 224)            3360      ['block4a_se_reduce[0][0]']   
                                                                                                  
 block4a_se_excite (Multipl  (None, 3, 3, 224)            0         ['block4a_activation[0][0]',  
 y)                                                                  'block4a_se_expand[0][0]']   
                                                                                                  
 block4a_project_conv (Conv  (None, 3, 3, 104)            23296     ['block4a_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block4a_project_bn (BatchN  (None, 3, 3, 104)            416       ['block4a_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block4b_expand_conv (Conv2  (None, 3, 3, 416)            43264     ['block4a_project_bn[0][0]']  
 D)                                                                                               
                                                                                                  
 block4b_expand_bn (BatchNo  (None, 3, 3, 416)            1664      ['block4b_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block4b_expand_activation   (None, 3, 3, 416)            0         ['block4b_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block4b_dwconv2 (Depthwise  (None, 3, 3, 416)            3744      ['block4b_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block4b_bn (BatchNormaliza  (None, 3, 3, 416)            1664      ['block4b_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block4b_activation (Activa  (None, 3, 3, 416)            0         ['block4b_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block4b_se_squeeze (Global  (None, 416)                  0         ['block4b_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block4b_se_reshape (Reshap  (None, 1, 1, 416)            0         ['block4b_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block4b_se_reduce (Conv2D)  (None, 1, 1, 26)             10842     ['block4b_se_reshape[0][0]']  
                                                                                                  
 block4b_se_expand (Conv2D)  (None, 1, 1, 416)            11232     ['block4b_se_reduce[0][0]']   
                                                                                                  
 block4b_se_excite (Multipl  (None, 3, 3, 416)            0         ['block4b_activation[0][0]',  
 y)                                                                  'block4b_se_expand[0][0]']   
                                                                                                  
 block4b_project_conv (Conv  (None, 3, 3, 104)            43264     ['block4b_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block4b_project_bn (BatchN  (None, 3, 3, 104)            416       ['block4b_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block4b_drop (Dropout)      (None, 3, 3, 104)            0         ['block4b_project_bn[0][0]']  
                                                                                                  
 block4b_add (Add)           (None, 3, 3, 104)            0         ['block4b_drop[0][0]',        
                                                                     'block4a_project_bn[0][0]']  
                                                                                                  
 block4c_expand_conv (Conv2  (None, 3, 3, 416)            43264     ['block4b_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block4c_expand_bn (BatchNo  (None, 3, 3, 416)            1664      ['block4c_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block4c_expand_activation   (None, 3, 3, 416)            0         ['block4c_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block4c_dwconv2 (Depthwise  (None, 3, 3, 416)            3744      ['block4c_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block4c_bn (BatchNormaliza  (None, 3, 3, 416)            1664      ['block4c_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block4c_activation (Activa  (None, 3, 3, 416)            0         ['block4c_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block4c_se_squeeze (Global  (None, 416)                  0         ['block4c_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block4c_se_reshape (Reshap  (None, 1, 1, 416)            0         ['block4c_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block4c_se_reduce (Conv2D)  (None, 1, 1, 26)             10842     ['block4c_se_reshape[0][0]']  
                                                                                                  
 block4c_se_expand (Conv2D)  (None, 1, 1, 416)            11232     ['block4c_se_reduce[0][0]']   
                                                                                                  
 block4c_se_excite (Multipl  (None, 3, 3, 416)            0         ['block4c_activation[0][0]',  
 y)                                                                  'block4c_se_expand[0][0]']   
                                                                                                  
 block4c_project_conv (Conv  (None, 3, 3, 104)            43264     ['block4c_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block4c_project_bn (BatchN  (None, 3, 3, 104)            416       ['block4c_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block4c_drop (Dropout)      (None, 3, 3, 104)            0         ['block4c_project_bn[0][0]']  
                                                                                                  
 block4c_add (Add)           (None, 3, 3, 104)            0         ['block4c_drop[0][0]',        
                                                                     'block4b_add[0][0]']         
                                                                                                  
 block4d_expand_conv (Conv2  (None, 3, 3, 416)            43264     ['block4c_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block4d_expand_bn (BatchNo  (None, 3, 3, 416)            1664      ['block4d_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block4d_expand_activation   (None, 3, 3, 416)            0         ['block4d_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block4d_dwconv2 (Depthwise  (None, 3, 3, 416)            3744      ['block4d_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block4d_bn (BatchNormaliza  (None, 3, 3, 416)            1664      ['block4d_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block4d_activation (Activa  (None, 3, 3, 416)            0         ['block4d_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block4d_se_squeeze (Global  (None, 416)                  0         ['block4d_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block4d_se_reshape (Reshap  (None, 1, 1, 416)            0         ['block4d_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block4d_se_reduce (Conv2D)  (None, 1, 1, 26)             10842     ['block4d_se_reshape[0][0]']  
                                                                                                  
 block4d_se_expand (Conv2D)  (None, 1, 1, 416)            11232     ['block4d_se_reduce[0][0]']   
                                                                                                  
 block4d_se_excite (Multipl  (None, 3, 3, 416)            0         ['block4d_activation[0][0]',  
 y)                                                                  'block4d_se_expand[0][0]']   
                                                                                                  
 block4d_project_conv (Conv  (None, 3, 3, 104)            43264     ['block4d_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block4d_project_bn (BatchN  (None, 3, 3, 104)            416       ['block4d_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block4d_drop (Dropout)      (None, 3, 3, 104)            0         ['block4d_project_bn[0][0]']  
                                                                                                  
 block4d_add (Add)           (None, 3, 3, 104)            0         ['block4d_drop[0][0]',        
                                                                     'block4c_add[0][0]']         
                                                                                                  
 block5a_expand_conv (Conv2  (None, 3, 3, 624)            64896     ['block4d_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block5a_expand_bn (BatchNo  (None, 3, 3, 624)            2496      ['block5a_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block5a_expand_activation   (None, 3, 3, 624)            0         ['block5a_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block5a_dwconv2 (Depthwise  (None, 3, 3, 624)            5616      ['block5a_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block5a_bn (BatchNormaliza  (None, 3, 3, 624)            2496      ['block5a_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block5a_activation (Activa  (None, 3, 3, 624)            0         ['block5a_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block5a_se_squeeze (Global  (None, 624)                  0         ['block5a_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block5a_se_reshape (Reshap  (None, 1, 1, 624)            0         ['block5a_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block5a_se_reduce (Conv2D)  (None, 1, 1, 26)             16250     ['block5a_se_reshape[0][0]']  
                                                                                                  
 block5a_se_expand (Conv2D)  (None, 1, 1, 624)            16848     ['block5a_se_reduce[0][0]']   
                                                                                                  
 block5a_se_excite (Multipl  (None, 3, 3, 624)            0         ['block5a_activation[0][0]',  
 y)                                                                  'block5a_se_expand[0][0]']   
                                                                                                  
 block5a_project_conv (Conv  (None, 3, 3, 120)            74880     ['block5a_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block5a_project_bn (BatchN  (None, 3, 3, 120)            480       ['block5a_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block5b_expand_conv (Conv2  (None, 3, 3, 720)            86400     ['block5a_project_bn[0][0]']  
 D)                                                                                               
                                                                                                  
 block5b_expand_bn (BatchNo  (None, 3, 3, 720)            2880      ['block5b_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block5b_expand_activation   (None, 3, 3, 720)            0         ['block5b_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block5b_dwconv2 (Depthwise  (None, 3, 3, 720)            6480      ['block5b_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block5b_bn (BatchNormaliza  (None, 3, 3, 720)            2880      ['block5b_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block5b_activation (Activa  (None, 3, 3, 720)            0         ['block5b_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block5b_se_squeeze (Global  (None, 720)                  0         ['block5b_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block5b_se_reshape (Reshap  (None, 1, 1, 720)            0         ['block5b_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block5b_se_reduce (Conv2D)  (None, 1, 1, 30)             21630     ['block5b_se_reshape[0][0]']  
                                                                                                  
 block5b_se_expand (Conv2D)  (None, 1, 1, 720)            22320     ['block5b_se_reduce[0][0]']   
                                                                                                  
 block5b_se_excite (Multipl  (None, 3, 3, 720)            0         ['block5b_activation[0][0]',  
 y)                                                                  'block5b_se_expand[0][0]']   
                                                                                                  
 block5b_project_conv (Conv  (None, 3, 3, 120)            86400     ['block5b_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block5b_project_bn (BatchN  (None, 3, 3, 120)            480       ['block5b_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block5b_drop (Dropout)      (None, 3, 3, 120)            0         ['block5b_project_bn[0][0]']  
                                                                                                  
 block5b_add (Add)           (None, 3, 3, 120)            0         ['block5b_drop[0][0]',        
                                                                     'block5a_project_bn[0][0]']  
                                                                                                  
 block5c_expand_conv (Conv2  (None, 3, 3, 720)            86400     ['block5b_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block5c_expand_bn (BatchNo  (None, 3, 3, 720)            2880      ['block5c_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block5c_expand_activation   (None, 3, 3, 720)            0         ['block5c_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block5c_dwconv2 (Depthwise  (None, 3, 3, 720)            6480      ['block5c_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block5c_bn (BatchNormaliza  (None, 3, 3, 720)            2880      ['block5c_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block5c_activation (Activa  (None, 3, 3, 720)            0         ['block5c_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block5c_se_squeeze (Global  (None, 720)                  0         ['block5c_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block5c_se_reshape (Reshap  (None, 1, 1, 720)            0         ['block5c_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block5c_se_reduce (Conv2D)  (None, 1, 1, 30)             21630     ['block5c_se_reshape[0][0]']  
                                                                                                  
 block5c_se_expand (Conv2D)  (None, 1, 1, 720)            22320     ['block5c_se_reduce[0][0]']   
                                                                                                  
 block5c_se_excite (Multipl  (None, 3, 3, 720)            0         ['block5c_activation[0][0]',  
 y)                                                                  'block5c_se_expand[0][0]']   
                                                                                                  
 block5c_project_conv (Conv  (None, 3, 3, 120)            86400     ['block5c_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block5c_project_bn (BatchN  (None, 3, 3, 120)            480       ['block5c_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block5c_drop (Dropout)      (None, 3, 3, 120)            0         ['block5c_project_bn[0][0]']  
                                                                                                  
 block5c_add (Add)           (None, 3, 3, 120)            0         ['block5c_drop[0][0]',        
                                                                     'block5b_add[0][0]']         
                                                                                                  
 block5d_expand_conv (Conv2  (None, 3, 3, 720)            86400     ['block5c_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block5d_expand_bn (BatchNo  (None, 3, 3, 720)            2880      ['block5d_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block5d_expand_activation   (None, 3, 3, 720)            0         ['block5d_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block5d_dwconv2 (Depthwise  (None, 3, 3, 720)            6480      ['block5d_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block5d_bn (BatchNormaliza  (None, 3, 3, 720)            2880      ['block5d_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block5d_activation (Activa  (None, 3, 3, 720)            0         ['block5d_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block5d_se_squeeze (Global  (None, 720)                  0         ['block5d_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block5d_se_reshape (Reshap  (None, 1, 1, 720)            0         ['block5d_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block5d_se_reduce (Conv2D)  (None, 1, 1, 30)             21630     ['block5d_se_reshape[0][0]']  
                                                                                                  
 block5d_se_expand (Conv2D)  (None, 1, 1, 720)            22320     ['block5d_se_reduce[0][0]']   
                                                                                                  
 block5d_se_excite (Multipl  (None, 3, 3, 720)            0         ['block5d_activation[0][0]',  
 y)                                                                  'block5d_se_expand[0][0]']   
                                                                                                  
 block5d_project_conv (Conv  (None, 3, 3, 120)            86400     ['block5d_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block5d_project_bn (BatchN  (None, 3, 3, 120)            480       ['block5d_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block5d_drop (Dropout)      (None, 3, 3, 120)            0         ['block5d_project_bn[0][0]']  
                                                                                                  
 block5d_add (Add)           (None, 3, 3, 120)            0         ['block5d_drop[0][0]',        
                                                                     'block5c_add[0][0]']         
                                                                                                  
 block5e_expand_conv (Conv2  (None, 3, 3, 720)            86400     ['block5d_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block5e_expand_bn (BatchNo  (None, 3, 3, 720)            2880      ['block5e_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block5e_expand_activation   (None, 3, 3, 720)            0         ['block5e_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block5e_dwconv2 (Depthwise  (None, 3, 3, 720)            6480      ['block5e_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block5e_bn (BatchNormaliza  (None, 3, 3, 720)            2880      ['block5e_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block5e_activation (Activa  (None, 3, 3, 720)            0         ['block5e_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block5e_se_squeeze (Global  (None, 720)                  0         ['block5e_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block5e_se_reshape (Reshap  (None, 1, 1, 720)            0         ['block5e_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block5e_se_reduce (Conv2D)  (None, 1, 1, 30)             21630     ['block5e_se_reshape[0][0]']  
                                                                                                  
 block5e_se_expand (Conv2D)  (None, 1, 1, 720)            22320     ['block5e_se_reduce[0][0]']   
                                                                                                  
 block5e_se_excite (Multipl  (None, 3, 3, 720)            0         ['block5e_activation[0][0]',  
 y)                                                                  'block5e_se_expand[0][0]']   
                                                                                                  
 block5e_project_conv (Conv  (None, 3, 3, 120)            86400     ['block5e_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block5e_project_bn (BatchN  (None, 3, 3, 120)            480       ['block5e_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block5e_drop (Dropout)      (None, 3, 3, 120)            0         ['block5e_project_bn[0][0]']  
                                                                                                  
 block5e_add (Add)           (None, 3, 3, 120)            0         ['block5e_drop[0][0]',        
                                                                     'block5d_add[0][0]']         
                                                                                                  
 block5f_expand_conv (Conv2  (None, 3, 3, 720)            86400     ['block5e_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block5f_expand_bn (BatchNo  (None, 3, 3, 720)            2880      ['block5f_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block5f_expand_activation   (None, 3, 3, 720)            0         ['block5f_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block5f_dwconv2 (Depthwise  (None, 3, 3, 720)            6480      ['block5f_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block5f_bn (BatchNormaliza  (None, 3, 3, 720)            2880      ['block5f_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block5f_activation (Activa  (None, 3, 3, 720)            0         ['block5f_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block5f_se_squeeze (Global  (None, 720)                  0         ['block5f_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block5f_se_reshape (Reshap  (None, 1, 1, 720)            0         ['block5f_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block5f_se_reduce (Conv2D)  (None, 1, 1, 30)             21630     ['block5f_se_reshape[0][0]']  
                                                                                                  
 block5f_se_expand (Conv2D)  (None, 1, 1, 720)            22320     ['block5f_se_reduce[0][0]']   
                                                                                                  
 block5f_se_excite (Multipl  (None, 3, 3, 720)            0         ['block5f_activation[0][0]',  
 y)                                                                  'block5f_se_expand[0][0]']   
                                                                                                  
 block5f_project_conv (Conv  (None, 3, 3, 120)            86400     ['block5f_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block5f_project_bn (BatchN  (None, 3, 3, 120)            480       ['block5f_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block5f_drop (Dropout)      (None, 3, 3, 120)            0         ['block5f_project_bn[0][0]']  
                                                                                                  
 block5f_add (Add)           (None, 3, 3, 120)            0         ['block5f_drop[0][0]',        
                                                                     'block5e_add[0][0]']         
                                                                                                  
 block6a_expand_conv (Conv2  (None, 3, 3, 720)            86400     ['block5f_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6a_expand_bn (BatchNo  (None, 3, 3, 720)            2880      ['block6a_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6a_expand_activation   (None, 3, 3, 720)            0         ['block6a_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6a_dwconv2 (Depthwise  (None, 2, 2, 720)            6480      ['block6a_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6a_bn (BatchNormaliza  (None, 2, 2, 720)            2880      ['block6a_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6a_activation (Activa  (None, 2, 2, 720)            0         ['block6a_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6a_se_squeeze (Global  (None, 720)                  0         ['block6a_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6a_se_reshape (Reshap  (None, 1, 1, 720)            0         ['block6a_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6a_se_reduce (Conv2D)  (None, 1, 1, 30)             21630     ['block6a_se_reshape[0][0]']  
                                                                                                  
 block6a_se_expand (Conv2D)  (None, 1, 1, 720)            22320     ['block6a_se_reduce[0][0]']   
                                                                                                  
 block6a_se_excite (Multipl  (None, 2, 2, 720)            0         ['block6a_activation[0][0]',  
 y)                                                                  'block6a_se_expand[0][0]']   
                                                                                                  
 block6a_project_conv (Conv  (None, 2, 2, 208)            149760    ['block6a_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6a_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6a_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6b_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6a_project_bn[0][0]']  
 D)                                                                                               
                                                                                                  
 block6b_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6b_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6b_expand_activation   (None, 2, 2, 1248)           0         ['block6b_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6b_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6b_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6b_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6b_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6b_activation (Activa  (None, 2, 2, 1248)           0         ['block6b_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6b_se_squeeze (Global  (None, 1248)                 0         ['block6b_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6b_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6b_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6b_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6b_se_reshape[0][0]']  
                                                                                                  
 block6b_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6b_se_reduce[0][0]']   
                                                                                                  
 block6b_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6b_activation[0][0]',  
 y)                                                                  'block6b_se_expand[0][0]']   
                                                                                                  
 block6b_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6b_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6b_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6b_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6b_drop (Dropout)      (None, 2, 2, 208)            0         ['block6b_project_bn[0][0]']  
                                                                                                  
 block6b_add (Add)           (None, 2, 2, 208)            0         ['block6b_drop[0][0]',        
                                                                     'block6a_project_bn[0][0]']  
                                                                                                  
 block6c_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6b_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6c_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6c_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6c_expand_activation   (None, 2, 2, 1248)           0         ['block6c_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6c_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6c_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6c_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6c_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6c_activation (Activa  (None, 2, 2, 1248)           0         ['block6c_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6c_se_squeeze (Global  (None, 1248)                 0         ['block6c_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6c_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6c_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6c_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6c_se_reshape[0][0]']  
                                                                                                  
 block6c_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6c_se_reduce[0][0]']   
                                                                                                  
 block6c_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6c_activation[0][0]',  
 y)                                                                  'block6c_se_expand[0][0]']   
                                                                                                  
 block6c_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6c_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6c_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6c_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6c_drop (Dropout)      (None, 2, 2, 208)            0         ['block6c_project_bn[0][0]']  
                                                                                                  
 block6c_add (Add)           (None, 2, 2, 208)            0         ['block6c_drop[0][0]',        
                                                                     'block6b_add[0][0]']         
                                                                                                  
 block6d_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6c_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6d_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6d_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6d_expand_activation   (None, 2, 2, 1248)           0         ['block6d_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6d_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6d_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6d_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6d_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6d_activation (Activa  (None, 2, 2, 1248)           0         ['block6d_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6d_se_squeeze (Global  (None, 1248)                 0         ['block6d_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6d_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6d_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6d_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6d_se_reshape[0][0]']  
                                                                                                  
 block6d_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6d_se_reduce[0][0]']   
                                                                                                  
 block6d_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6d_activation[0][0]',  
 y)                                                                  'block6d_se_expand[0][0]']   
                                                                                                  
 block6d_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6d_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6d_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6d_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6d_drop (Dropout)      (None, 2, 2, 208)            0         ['block6d_project_bn[0][0]']  
                                                                                                  
 block6d_add (Add)           (None, 2, 2, 208)            0         ['block6d_drop[0][0]',        
                                                                     'block6c_add[0][0]']         
                                                                                                  
 block6e_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6d_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6e_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6e_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6e_expand_activation   (None, 2, 2, 1248)           0         ['block6e_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6e_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6e_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6e_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6e_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6e_activation (Activa  (None, 2, 2, 1248)           0         ['block6e_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6e_se_squeeze (Global  (None, 1248)                 0         ['block6e_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6e_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6e_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6e_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6e_se_reshape[0][0]']  
                                                                                                  
 block6e_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6e_se_reduce[0][0]']   
                                                                                                  
 block6e_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6e_activation[0][0]',  
 y)                                                                  'block6e_se_expand[0][0]']   
                                                                                                  
 block6e_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6e_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6e_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6e_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6e_drop (Dropout)      (None, 2, 2, 208)            0         ['block6e_project_bn[0][0]']  
                                                                                                  
 block6e_add (Add)           (None, 2, 2, 208)            0         ['block6e_drop[0][0]',        
                                                                     'block6d_add[0][0]']         
                                                                                                  
 block6f_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6e_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6f_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6f_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6f_expand_activation   (None, 2, 2, 1248)           0         ['block6f_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6f_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6f_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6f_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6f_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6f_activation (Activa  (None, 2, 2, 1248)           0         ['block6f_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6f_se_squeeze (Global  (None, 1248)                 0         ['block6f_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6f_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6f_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6f_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6f_se_reshape[0][0]']  
                                                                                                  
 block6f_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6f_se_reduce[0][0]']   
                                                                                                  
 block6f_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6f_activation[0][0]',  
 y)                                                                  'block6f_se_expand[0][0]']   
                                                                                                  
 block6f_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6f_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6f_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6f_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6f_drop (Dropout)      (None, 2, 2, 208)            0         ['block6f_project_bn[0][0]']  
                                                                                                  
 block6f_add (Add)           (None, 2, 2, 208)            0         ['block6f_drop[0][0]',        
                                                                     'block6e_add[0][0]']         
                                                                                                  
 block6g_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6f_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6g_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6g_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6g_expand_activation   (None, 2, 2, 1248)           0         ['block6g_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6g_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6g_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6g_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6g_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6g_activation (Activa  (None, 2, 2, 1248)           0         ['block6g_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6g_se_squeeze (Global  (None, 1248)                 0         ['block6g_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6g_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6g_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6g_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6g_se_reshape[0][0]']  
                                                                                                  
 block6g_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6g_se_reduce[0][0]']   
                                                                                                  
 block6g_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6g_activation[0][0]',  
 y)                                                                  'block6g_se_expand[0][0]']   
                                                                                                  
 block6g_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6g_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6g_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6g_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6g_drop (Dropout)      (None, 2, 2, 208)            0         ['block6g_project_bn[0][0]']  
                                                                                                  
 block6g_add (Add)           (None, 2, 2, 208)            0         ['block6g_drop[0][0]',        
                                                                     'block6f_add[0][0]']         
                                                                                                  
 block6h_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6g_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6h_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6h_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6h_expand_activation   (None, 2, 2, 1248)           0         ['block6h_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6h_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6h_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6h_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6h_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6h_activation (Activa  (None, 2, 2, 1248)           0         ['block6h_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6h_se_squeeze (Global  (None, 1248)                 0         ['block6h_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6h_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6h_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6h_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6h_se_reshape[0][0]']  
                                                                                                  
 block6h_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6h_se_reduce[0][0]']   
                                                                                                  
 block6h_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6h_activation[0][0]',  
 y)                                                                  'block6h_se_expand[0][0]']   
                                                                                                  
 block6h_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6h_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6h_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6h_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6h_drop (Dropout)      (None, 2, 2, 208)            0         ['block6h_project_bn[0][0]']  
                                                                                                  
 block6h_add (Add)           (None, 2, 2, 208)            0         ['block6h_drop[0][0]',        
                                                                     'block6g_add[0][0]']         
                                                                                                  
 block6i_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6h_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6i_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6i_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6i_expand_activation   (None, 2, 2, 1248)           0         ['block6i_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6i_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6i_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6i_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6i_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6i_activation (Activa  (None, 2, 2, 1248)           0         ['block6i_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6i_se_squeeze (Global  (None, 1248)                 0         ['block6i_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6i_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6i_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6i_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6i_se_reshape[0][0]']  
                                                                                                  
 block6i_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6i_se_reduce[0][0]']   
                                                                                                  
 block6i_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6i_activation[0][0]',  
 y)                                                                  'block6i_se_expand[0][0]']   
                                                                                                  
 block6i_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6i_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6i_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6i_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6i_drop (Dropout)      (None, 2, 2, 208)            0         ['block6i_project_bn[0][0]']  
                                                                                                  
 block6i_add (Add)           (None, 2, 2, 208)            0         ['block6i_drop[0][0]',        
                                                                     'block6h_add[0][0]']         
                                                                                                  
 block6j_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6i_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6j_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6j_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6j_expand_activation   (None, 2, 2, 1248)           0         ['block6j_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6j_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6j_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6j_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6j_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6j_activation (Activa  (None, 2, 2, 1248)           0         ['block6j_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6j_se_squeeze (Global  (None, 1248)                 0         ['block6j_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6j_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6j_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6j_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6j_se_reshape[0][0]']  
                                                                                                  
 block6j_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6j_se_reduce[0][0]']   
                                                                                                  
 block6j_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6j_activation[0][0]',  
 y)                                                                  'block6j_se_expand[0][0]']   
                                                                                                  
 block6j_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6j_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6j_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6j_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6j_drop (Dropout)      (None, 2, 2, 208)            0         ['block6j_project_bn[0][0]']  
                                                                                                  
 block6j_add (Add)           (None, 2, 2, 208)            0         ['block6j_drop[0][0]',        
                                                                     'block6i_add[0][0]']         
                                                                                                  
 top_conv (Conv2D)           (None, 2, 2, 1408)           292864    ['block6j_add[0][0]']         
                                                                                                  
 top_bn (BatchNormalization  (None, 2, 2, 1408)           5632      ['top_conv[0][0]']            
 )                                                                                                
                                                                                                  
 top_activation (Activation  (None, 2, 2, 1408)           0         ['top_bn[0][0]']              
 )                                                                                                
                                                                                                  
==================================================================================================
Total params: 8769374 (33.45 MB)
Trainable params: 8687086 (33.14 MB)
Non-trainable params: 82288 (321.44 KB)
__________________________________________________________________________________________________

# Visualize the model as blocks
plot_model(EfficientNet, to_file='model.png', show_shapes=True, show_layer_names=True)

Model Building

# Set ResNet layers to non-trainable
EfficientNet.trainable = False

# Rebuild the model
x = EfficientNet.output
# Add Global Average Pooling layer
x = GlobalAveragePooling2D()(x)

# Flatten the output
x = Flatten()(x)

# Add a Dense layer (optional: you can add BatchNormalization and/or Dropout here)
x = Dense(256, activation='relu')(x)
x = BatchNormalization()(x)  # Optional
x = Dropout(0.3)(x)          # Optional

# Add the final Dense layer with 4 neurons and a softmax activation function
pred = Dense(4, activation='softmax')(x)

# Initializing the model
Efficientnetmodel = Model(inputs=EfficientNet.input, outputs=pred)

Compiling and Training the Model

# ModelCheckpoint is used to save the model at certain intervals
checkpoint_efnet = ModelCheckpoint("efnet.h5", # file path where the model will be saved
                    monitor='val_accuracy',  # means the callback monitors the validation accuracy
                    verbose=1,               # model is saved only when the metric (val_acc) has improved
                    save_best_only=True,
                    mode='max')

# Configured callbacks will be passed to the model during training
callbacks_list_efnet = [early_stopping, checkpoint_efnet, reduce_learningrate]

# Number of iterations over the entire dataset that the training process should run
epochs = 20

# Compile the model (This step is required if you're about to train the model)
Efficientnetmodel.compile(loss='categorical_crossentropy',
                          optimizer='adam',
                          metrics=['accuracy'])

# Fitting the model with required parameters above, saving model into history_efnet variable
history_efnet = Efficientnetmodel.fit(
    train_set_pre,
    steps_per_epoch = train_set_pre.samples // train_set_pre.batch_size,
    validation_data = validation_set_pre,
    validation_steps = validation_set_pre.samples // validation_set_pre.batch_size,
    epochs = 20,
    verbose = 1,
    callbacks = callbacks_list_efnet
)

Epoch 1/20
472/472 [==============================] - ETA: 0s - loss: 1.4951 - accuracy: 0.2569
Epoch 1: val_accuracy improved from -inf to 0.24435, saving model to efnet.h5
472/472 [==============================] - 27s 52ms/step - loss: 1.4951 - accuracy: 0.2569 - val_loss: 1.7732 - val_accuracy: 0.2444 - lr: 0.0010
Epoch 2/20
472/472 [==============================] - ETA: 0s - loss: 1.4112 - accuracy: 0.2630
Epoch 2: val_accuracy improved from 0.24435 to 0.36673, saving model to efnet.h5
472/472 [==============================] - 24s 52ms/step - loss: 1.4112 - accuracy: 0.2630 - val_loss: 1.3607 - val_accuracy: 0.3667 - lr: 0.0010
Epoch 3/20
471/472 [============================>.] - ETA: 0s - loss: 1.3959 - accuracy: 0.2631
Epoch 3: val_accuracy did not improve from 0.36673
472/472 [==============================] - 23s 50ms/step - loss: 1.3960 - accuracy: 0.2629 - val_loss: 1.3424 - val_accuracy: 0.3665 - lr: 0.0010
Epoch 4/20
471/472 [============================>.] - ETA: 0s - loss: 1.3921 - accuracy: 0.2635
Epoch 4: val_accuracy did not improve from 0.36673
472/472 [==============================] - 23s 49ms/step - loss: 1.3921 - accuracy: 0.2636 - val_loss: 1.3576 - val_accuracy: 0.3163 - lr: 0.0010
Epoch 5/20
472/472 [==============================] - ETA: 0s - loss: 1.3906 - accuracy: 0.2663
Epoch 5: val_accuracy improved from 0.36673 to 0.36734, saving model to efnet.h5
472/472 [==============================] - 24s 50ms/step - loss: 1.3906 - accuracy: 0.2663 - val_loss: 1.3653 - val_accuracy: 0.3673 - lr: 0.0010
Epoch 6/20
471/472 [============================>.] - ETA: 0s - loss: 1.3911 - accuracy: 0.2611Restoring model weights from the end of the best epoch: 3.

Epoch 6: val_accuracy did not improve from 0.36734

Epoch 6: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026.
472/472 [==============================] - 23s 50ms/step - loss: 1.3910 - accuracy: 0.2614 - val_loss: 1.3712 - val_accuracy: 0.2599 - lr: 0.0010
Epoch 6: early stopping

Evaluating the EfficientnetNet Model

# Load the saved model if notebook session gets disconnected
Efficientnetmodel = load_model("efnet.h5")

# Evaluate the model on the test data
test_loss, test_accuracy = Efficientnetmodel.evaluate(test_set_pre, verbose=1)

# Print the results
print("Test Loss:", test_loss)
print("Test Accuracy:", test_accuracy)

4/4 [==============================] - 0s 52ms/step - loss: 1.4318 - accuracy: 0.2500
Test Loss: 1.4318348169326782
Test Accuracy: 0.25

# Plotting the Training and Validation accuracies of EfficientNet model
plt.plot(history_efnet.history['accuracy'])
plt.plot(history_efnet.history['val_accuracy'])
plt.title('EfficientNet Transfer Learning Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc = 'upper left')

# Display the plot
plt.show()

Observations and Insights:

• EfficientNet model has performed worse in both training and validation than any other model tried in this notebook,but much worse when compared to our custom CNN model for Greyscale images previously chosen.
  • Chosen CNN for Greyscale images: 71.1% training accuracy, 68% validation accuracy.
  • VGG16: 26.1% training accuracy, 26% validation accuracy.
• Again, training has stopped very prematurely due to Learning Rate platoeing, which might indicate early overfitting from a model that does not generalises enough for this task. Perhaps relaxing the conditions at which the condition is met might allow the model to improve its performance over time or could lead to further overfitting.

EfficientNet model hasn't performed better than our selected CNN Model for Greyscale images. An early stopping was triggered at a very early epoch 6, which might have been due to the model not generalising enough for the task. Considering the above, the CNN Model for Greyscale images is still the best performing model thus far.

Conclusion

• The custom CNN model for grayscale images significantly outperforms the transfer learning models (VGG16, ResnetV2, and EfficientNet) in both training and validation accuracies.
• Custom CNN for Greyscale images: 71.1% training accuracy, 68% validation accuracy.
• VGG16: 50.2% training accuracy, 47.4% validation accuracy.
• ResnetV2: 38.2% training accuracy, 30.75% validation accuracy.
• EfficientNet: 26.1% training accuracy, 26% validation accuracy.

Early stopping due to plateauing learning rates in these models suggests that they are not learning effectively from the dataset, possibly due to over-specialization on their original training data and not adapting well to the new task.

It seems like the Pre-trained models are too complex for the task at hands and do not generalise enough, which resulted in poor performance when compared to a custom CNN model.

In conclusion, the comparative underperformance of the transfer learning models in this task highlights the importance of aligning model complexity and architecture with the specific characteristics of the dataset and the task at hand. The 'rgb' color mode, while generally beneficial for color images, seems to be less effective for grayscale image datasets, as seen in this case.

Building a Complex Neural Network Architecture

In this section, we will build a more complex Convolutional Neural Network Model that has close to as many parameters as we had in our Transfer Learning Models. However, we will have only 1 input channel for our input images.

Creating our Data Loaders

In this section, we are creating data loaders which we will use as inputs to the more Complicated Convolutional Neural Network. We will go ahead with color_mode = 'grayscale'.

# Clearing backend
backend.clear_session()

# Reset seed for number generators using reset_seed() function
reset_seed(fixed_seed)

Random number generators seed has been set to 42

# Creating dataloader for train for Greyscale dataset
train_set_gray2 = datagen.flow_from_directory(folder_path + "train",
                                        target_size = (img_size),
                                        color_mode = 'grayscale',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        classes = ['happy', 'sad', 'neutral', 'surprise'],
                                        shuffle = True)

# Creating dataloader for validation for Greyscale dataset
validation_set_gray2 = datagen.flow_from_directory(folder_path + "validation",
                                        target_size = (img_size),
                                        color_mode = 'grayscale',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        classes = ['happy', 'sad', 'neutral', 'surprise'],
                                        shuffle = True)

# Creating dataloader for test for Greyscale dataset
test_set_gray2 = datagen.flow_from_directory(folder_path + "test",
                                        target_size = (img_size),
                                        color_mode = 'grayscale',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        classes = ['happy', 'sad', 'neutral', 'surprise'],
                                        shuffle = True)

Found 15109 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.
Found 128 images belonging to 4 classes.

Model Building

• Try building a layer with 5 Convolutional Blocks and see if performance increases.

from keras.layers import Dense, BatchNormalization, Activation, Dropout

# Define more complex CNN model
def complex_cnn():
    cnn_model = Sequential()

    # Building a layer with 5 Convolutional Blocks and see if performance increases
    # Add first Conv2D layer with 64 filters and a kernel size of 2. Use the 'same'
    # padding and provide the input shape = (48, 48, 1). Use 'relu' activation.
    cnn_model.add(Conv2D(filters = 64, 
                         kernel_size = (2, 2), 
                         padding = "same", 
                         input_shape = (img_size[0], img_size[1], 1), activation = 'relu'))
    # Adding a BatchNormalization layer
    cnn_model.add(BatchNormalization())
    # LeakyRelU layer with Leaky ReLU parameter of 0.1
    cnn_model.add(LeakyReLU(0.1))
    # A max-pooling layer with a pool size of 2x2
    cnn_model.add(MaxPooling2D(pool_size = (2, 2)))
    # Dropout layer with the rate equal to 0.2
    cnn_model.add(Dropout(0.2))
    # Second Convolutional layer with 128 filters and the kernel size of 2x2 with 'same' padding
    cnn_model.add(Conv2D(filters = 128, kernel_size = (2, 2), padding = "same", activation = 'relu'))
    # Adding a BatchNormalization layer
    cnn_model.add(BatchNormalization())
    # LeakyRelU layer with Leaky ReLU parameter of 0.1
    cnn_model.add(LeakyReLU(0.1))
    # A max-pooling layer with a pool size of 2x2
    cnn_model.add(MaxPooling2D(pool_size = (2, 2)))
    # Dropout layer with the rate equal to 0.2
    cnn_model.add(Dropout(0.2))
    # Third Convolutional layer with 512 filters and the kernel size of 2x2 with 'same' padding
    cnn_model.add(Conv2D(filters = 512, kernel_size = (2, 2), padding = "same", activation = 'relu'))
    # Adding a BatchNormalization layer
    cnn_model.add(BatchNormalization())
    # LeakyRelU layer with Leaky ReLU parameter of 0.1
    cnn_model.add(LeakyReLU(0.1))
    # A max-pooling layer with a pool size of 2x2
    cnn_model.add(MaxPooling2D(pool_size = (2, 2)))
    # Dropout layer with the rate equal to 0.2
    cnn_model.add(Dropout(0.2))
    # Fourth Convolutional layer with 512 filters and the kernel size of 2x2 with 'same' padding
    cnn_model.add(Conv2D(filters = 512, kernel_size = (2, 2), padding = "same", activation = 'relu'))
    # Adding a BatchNormalization layer
    cnn_model.add(BatchNormalization())
    # LeakyRelU layer with Leaky ReLU parameter of 0.1
    cnn_model.add(LeakyReLU(0.1))
    # A max-pooling layer with a pool size of 2x2
    cnn_model.add(MaxPooling2D(pool_size = (2, 2)))
    # Dropout layer with the rate equal to 0.2
    cnn_model.add(Dropout(0.2))
    # Fifth Convolutional layer with 128 filters and the kernel size of 2x2 with 'same' padding
    cnn_model.add(Conv2D(filters = 128, kernel_size = (2, 2), padding = "same", activation = 'relu'))
    # Adding a BatchNormalization layer
    cnn_model.add(BatchNormalization())
    # LeakyRelU layer with Leaky ReLU parameter of 0.1
    cnn_model.add(LeakyReLU(0.1))
    # A max-pooling layer with a pool size of 2x2
    cnn_model.add(MaxPooling2D(pool_size = (2, 2)))
    # Dropout layer with the rate equal to 0.2
    cnn_model.add(Dropout(0.2))
    # Flatten the output from the previous layer
    cnn_model.add(Flatten())
    # Dense layer with 256 nodes
    cnn_model.add(Dense(256, activation = 'relu'))
    # Adding a BatchNormalization layer
    cnn_model.add(BatchNormalization())
    # Adding a 'relu' Activation layer
    cnn_model.add(Activation('relu'))
    # Dropout layer with the rate equal to 0.2
    cnn_model.add(Dropout(0.2))
    # Dense layer with 512 nodes
    cnn_model.add(Dense(512, activation = 'relu'))
    # Adding a BatchNormalization layer
    cnn_model.add(BatchNormalization())
    # Adding a 'relu' Activation layer
    cnn_model.add(Activation('relu'))
    # Dropout layer with the rate equal to 0.2
    cnn_model.add(Dropout(0.2))
    # Final output layer with nodes equal to the number of classes (sad, happy, neutral and surprised) and 'softmax' as the activation function
    cnn_model.add(Dense(4, activation = 'softmax'))
    # Compiling the CNN model with categorical crossentropy as loss function, Adam Optimizer
    # with 0.001 learning rate, and set metrics set to 'accuracy'.
    cnn_model.compile(loss='categorical_crossentropy',
              optimizer=legacy.Adam(learning_rate = 0.001),
              metrics=['accuracy'])
    return cnn_model

# Building the model
complex_cnn_model = complex_cnn()
# Print first model's summary
complex_cnn_model.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 64)        320       
                                                                 
 batch_normalization (Batch  (None, 48, 48, 64)        256       
 Normalization)                                                  
                                                                 
 leaky_re_lu (LeakyReLU)     (None, 48, 48, 64)        0         
                                                                 
 max_pooling2d (MaxPooling2  (None, 24, 24, 64)        0         
 D)                                                              
                                                                 
 dropout (Dropout)           (None, 24, 24, 64)        0         
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 128)       32896     
                                                                 
 batch_normalization_1 (Bat  (None, 24, 24, 128)       512       
 chNormalization)                                                
                                                                 
 leaky_re_lu_1 (LeakyReLU)   (None, 24, 24, 128)       0         
                                                                 
 max_pooling2d_1 (MaxPoolin  (None, 12, 12, 128)       0         
 g2D)                                                            
                                                                 
 dropout_1 (Dropout)         (None, 12, 12, 128)       0         
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 512)       262656    
                                                                 
 batch_normalization_2 (Bat  (None, 12, 12, 512)       2048      
 chNormalization)                                                
                                                                 
 leaky_re_lu_2 (LeakyReLU)   (None, 12, 12, 512)       0         
                                                                 
 max_pooling2d_2 (MaxPoolin  (None, 6, 6, 512)         0         
 g2D)                                                            
                                                                 
 dropout_2 (Dropout)         (None, 6, 6, 512)         0         
                                                                 
 conv2d_3 (Conv2D)           (None, 6, 6, 512)         1049088   
                                                                 
 batch_normalization_3 (Bat  (None, 6, 6, 512)         2048      
 chNormalization)                                                
                                                                 
 leaky_re_lu_3 (LeakyReLU)   (None, 6, 6, 512)         0         
                                                                 
 max_pooling2d_3 (MaxPoolin  (None, 3, 3, 512)         0         
 g2D)                                                            
                                                                 
 dropout_3 (Dropout)         (None, 3, 3, 512)         0         
                                                                 
 conv2d_4 (Conv2D)           (None, 3, 3, 128)         262272    
                                                                 
 batch_normalization_4 (Bat  (None, 3, 3, 128)         512       
 chNormalization)                                                
                                                                 
 leaky_re_lu_4 (LeakyReLU)   (None, 3, 3, 128)         0         
                                                                 
 max_pooling2d_4 (MaxPoolin  (None, 1, 1, 128)         0         
 g2D)                                                            
                                                                 
 dropout_4 (Dropout)         (None, 1, 1, 128)         0         
                                                                 
 flatten (Flatten)           (None, 128)               0         
                                                                 
 dense (Dense)               (None, 256)               33024     
                                                                 
 batch_normalization_5 (Bat  (None, 256)               1024      
 chNormalization)                                                
                                                                 
 activation (Activation)     (None, 256)               0         
                                                                 
 dropout_5 (Dropout)         (None, 256)               0         
                                                                 
 dense_1 (Dense)             (None, 512)               131584    
                                                                 
 batch_normalization_6 (Bat  (None, 512)               2048      
 chNormalization)                                                
                                                                 
 activation_1 (Activation)   (None, 512)               0         
                                                                 
 dropout_6 (Dropout)         (None, 512)               0         
                                                                 
 dense_2 (Dense)             (None, 4)                 2052      
                                                                 
=================================================================
Total params: 1782340 (6.80 MB)
Trainable params: 1778116 (6.78 MB)
Non-trainable params: 4224 (16.50 KB)
_________________________________________________________________

# Visualize the model as blocks
plot_model(complex_cnn_model, to_file='model.png', show_shapes=True, show_layer_names=True)

Compiling and Training the Model

# ModelCheckpoint is used to save the model at certain intervals
checkpoint_complex_cnn = ModelCheckpoint("complex_cnn.h5", # file path where the model will be saved
                             monitor='val_accuracy',  # means the callback monitors the validation accuracy
                             verbose=1,               # model is saved only when the metric (val_acc) has improved
                             save_best_only=True,
                             mode='max')

# Configured callbacks will be passed to the model during training
callbacks_list_complex_cnn = [early_stopping, checkpoint_complex_cnn, reduce_learningrate]

# Number of iterations over the entire dataset that the training process should run
epochs = 35

# Compiling the Complex CNN model with categorical crossentropy as loss function, Adam Optimizer
# with 0.003 learning rate, and set metrics set to 'accuracy'.
complex_cnn_model.compile(loss='categorical_crossentropy',
          optimizer=legacy.Adam(learning_rate = 0.003),
          metrics=['accuracy'])

# Fitting the model with required parameters above, saving model into history_complex_cnn variable
history_complex_cnn = complex_cnn_model.fit(
    train_set_gray2,
    steps_per_epoch = train_set_gray2.samples // train_set_gray2.batch_size,
    validation_data = validation_set_gray2,
    validation_steps = validation_set_gray2.samples // validation_set_gray2.batch_size,
    epochs = 20,
    verbose = 1,
    callbacks = callbacks_list_complex_cnn
)

Epoch 1/20
472/472 [==============================] - ETA: 0s - loss: 1.5206 - accuracy: 0.3041
Epoch 1: val_accuracy improved from -inf to 0.22903, saving model to complex_cnn.h5
472/472 [==============================] - 1672s 4s/step - loss: 1.5206 - accuracy: 0.3041 - val_loss: 1.4122 - val_accuracy: 0.2290 - lr: 0.0030
Epoch 2/20
472/472 [==============================] - ETA: 0s - loss: 1.3079 - accuracy: 0.3838
Epoch 2: val_accuracy improved from 0.22903 to 0.38387, saving model to complex_cnn.h5
472/472 [==============================] - 1661s 4s/step - loss: 1.3079 - accuracy: 0.3838 - val_loss: 1.6416 - val_accuracy: 0.3839 - lr: 0.0030
Epoch 3/20
472/472 [==============================] - ETA: 0s - loss: 1.2012 - accuracy: 0.4497
Epoch 3: val_accuracy did not improve from 0.38387
472/472 [==============================] - 1687s 4s/step - loss: 1.2012 - accuracy: 0.4497 - val_loss: 1.4566 - val_accuracy: 0.3149 - lr: 0.0030
Epoch 4/20
472/472 [==============================] - ETA: 0s - loss: 1.1247 - accuracy: 0.4902
Epoch 4: val_accuracy improved from 0.38387 to 0.48589, saving model to complex_cnn.h5
472/472 [==============================] - 1657s 4s/step - loss: 1.1247 - accuracy: 0.4902 - val_loss: 1.1265 - val_accuracy: 0.4859 - lr: 0.0030
Epoch 5/20
472/472 [==============================] - ETA: 0s - loss: 1.0590 - accuracy: 0.5402
Epoch 5: val_accuracy improved from 0.48589 to 0.54657, saving model to complex_cnn.h5
472/472 [==============================] - 1650s 3s/step - loss: 1.0590 - accuracy: 0.5402 - val_loss: 1.0444 - val_accuracy: 0.5466 - lr: 0.0030
Epoch 6/20
472/472 [==============================] - ETA: 0s - loss: 1.0188 - accuracy: 0.5548
Epoch 6: val_accuracy did not improve from 0.54657
472/472 [==============================] - 1628s 3s/step - loss: 1.0188 - accuracy: 0.5548 - val_loss: 1.7051 - val_accuracy: 0.4889 - lr: 0.0030
Epoch 7/20
472/472 [==============================] - ETA: 0s - loss: 0.9788 - accuracy: 0.5744
Epoch 7: val_accuracy improved from 0.54657 to 0.56754, saving model to complex_cnn.h5
472/472 [==============================] - 1624s 3s/step - loss: 0.9788 - accuracy: 0.5744 - val_loss: 0.9949 - val_accuracy: 0.5675 - lr: 0.0030
Epoch 8/20
472/472 [==============================] - ETA: 0s - loss: 0.9385 - accuracy: 0.5986
Epoch 8: val_accuracy improved from 0.56754 to 0.64718, saving model to complex_cnn.h5
472/472 [==============================] - 1662s 4s/step - loss: 0.9385 - accuracy: 0.5986 - val_loss: 1.2303 - val_accuracy: 0.6472 - lr: 0.0030
Epoch 9/20
472/472 [==============================] - ETA: 0s - loss: 0.9204 - accuracy: 0.6058
Epoch 9: val_accuracy did not improve from 0.64718
472/472 [==============================] - 1563s 3s/step - loss: 0.9204 - accuracy: 0.6058 - val_loss: 1.7770 - val_accuracy: 0.5704 - lr: 0.0030
Epoch 10/20
472/472 [==============================] - ETA: 0s - loss: 0.8911 - accuracy: 0.6192
Epoch 10: val_accuracy improved from 0.64718 to 0.64879, saving model to complex_cnn.h5
472/472 [==============================] - 1527s 3s/step - loss: 0.8911 - accuracy: 0.6192 - val_loss: 0.8410 - val_accuracy: 0.6488 - lr: 0.0030
Epoch 11/20
472/472 [==============================] - ETA: 0s - loss: 0.8497 - accuracy: 0.6406
Epoch 11: val_accuracy did not improve from 0.64879
472/472 [==============================] - 1531s 3s/step - loss: 0.8497 - accuracy: 0.6406 - val_loss: 1.2406 - val_accuracy: 0.5897 - lr: 0.0030
Epoch 12/20
472/472 [==============================] - ETA: 0s - loss: 0.8294 - accuracy: 0.6520
Epoch 12: val_accuracy improved from 0.64879 to 0.65706, saving model to complex_cnn.h5
472/472 [==============================] - 1544s 3s/step - loss: 0.8294 - accuracy: 0.6520 - val_loss: 0.8466 - val_accuracy: 0.6571 - lr: 0.0030
Epoch 13/20
472/472 [==============================] - ETA: 0s - loss: 0.7901 - accuracy: 0.6725
Epoch 13: val_accuracy improved from 0.65706 to 0.67823, saving model to complex_cnn.h5
472/472 [==============================] - 1531s 3s/step - loss: 0.7901 - accuracy: 0.6725 - val_loss: 0.8096 - val_accuracy: 0.6782 - lr: 0.0030
Epoch 14/20
472/472 [==============================] - ETA: 0s - loss: 0.8355 - accuracy: 0.6467
Epoch 14: val_accuracy did not improve from 0.67823
472/472 [==============================] - 1540s 3s/step - loss: 0.8355 - accuracy: 0.6467 - val_loss: 1.5303 - val_accuracy: 0.5716 - lr: 0.0030
Epoch 15/20
472/472 [==============================] - ETA: 0s - loss: 0.8029 - accuracy: 0.6600
Epoch 15: val_accuracy did not improve from 0.67823
472/472 [==============================] - 1517s 3s/step - loss: 0.8029 - accuracy: 0.6600 - val_loss: 1.1180 - val_accuracy: 0.6544 - lr: 0.0030
Epoch 16/20
472/472 [==============================] - ETA: 0s - loss: 0.8643 - accuracy: 0.6320Restoring model weights from the end of the best epoch: 13.

Epoch 16: val_accuracy did not improve from 0.67823

Epoch 16: ReduceLROnPlateau reducing learning rate to 0.0006000000052154065.
472/472 [==============================] - 1523s 3s/step - loss: 0.8643 - accuracy: 0.6320 - val_loss: 0.8522 - val_accuracy: 0.6417 - lr: 0.0030
Epoch 16: early stopping

Evaluating the Model on Test Set

# Load the saved model if notebook session gets disconnected
complex_cnn_model = load_model("complex_cnn.h5")

# Print first model's summary
complex_cnn_model.summary()

Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d_5 (Conv2D)           (None, 224, 224, 64)      320       
                                                                 
 batch_normalization_7 (Bat  (None, 224, 224, 64)      256       
 chNormalization)                                                
                                                                 
 leaky_re_lu_5 (LeakyReLU)   (None, 224, 224, 64)      0         
                                                                 
 max_pooling2d_5 (MaxPoolin  (None, 112, 112, 64)      0         
 g2D)                                                            
                                                                 
 dropout_7 (Dropout)         (None, 112, 112, 64)      0         
                                                                 
 conv2d_6 (Conv2D)           (None, 112, 112, 128)     32896     
                                                                 
 batch_normalization_8 (Bat  (None, 112, 112, 128)     512       
 chNormalization)                                                
                                                                 
 leaky_re_lu_6 (LeakyReLU)   (None, 112, 112, 128)     0         
                                                                 
 max_pooling2d_6 (MaxPoolin  (None, 56, 56, 128)       0         
 g2D)                                                            
                                                                 
 dropout_8 (Dropout)         (None, 56, 56, 128)       0         
                                                                 
 conv2d_7 (Conv2D)           (None, 56, 56, 512)       262656    
                                                                 
 batch_normalization_9 (Bat  (None, 56, 56, 512)       2048      
 chNormalization)                                                
                                                                 
 leaky_re_lu_7 (LeakyReLU)   (None, 56, 56, 512)       0         
                                                                 
 max_pooling2d_7 (MaxPoolin  (None, 28, 28, 512)       0         
 g2D)                                                            
                                                                 
 dropout_9 (Dropout)         (None, 28, 28, 512)       0         
                                                                 
 conv2d_8 (Conv2D)           (None, 28, 28, 512)       1049088   
                                                                 
 batch_normalization_10 (Ba  (None, 28, 28, 512)       2048      
 tchNormalization)                                               
                                                                 
 leaky_re_lu_8 (LeakyReLU)   (None, 28, 28, 512)       0         
                                                                 
 max_pooling2d_8 (MaxPoolin  (None, 14, 14, 512)       0         
 g2D)                                                            
                                                                 
 dropout_10 (Dropout)        (None, 14, 14, 512)       0         
                                                                 
 conv2d_9 (Conv2D)           (None, 14, 14, 128)       262272    
                                                                 
 batch_normalization_11 (Ba  (None, 14, 14, 128)       512       
 tchNormalization)                                               
                                                                 
 leaky_re_lu_9 (LeakyReLU)   (None, 14, 14, 128)       0         
                                                                 
 max_pooling2d_9 (MaxPoolin  (None, 7, 7, 128)         0         
 g2D)                                                            
                                                                 
 dropout_11 (Dropout)        (None, 7, 7, 128)         0         
                                                                 
 flatten_1 (Flatten)         (None, 6272)              0         
                                                                 
 dense_3 (Dense)             (None, 256)               1605888   
                                                                 
 batch_normalization_12 (Ba  (None, 256)               1024      
 tchNormalization)                                               
                                                                 
 activation_2 (Activation)   (None, 256)               0         
                                                                 
 dropout_12 (Dropout)        (None, 256)               0         
                                                                 
 dense_4 (Dense)             (None, 512)               131584    
                                                                 
 batch_normalization_13 (Ba  (None, 512)               2048      
 tchNormalization)                                               
                                                                 
 activation_3 (Activation)   (None, 512)               0         
                                                                 
 dropout_13 (Dropout)        (None, 512)               0         
                                                                 
 dense_5 (Dense)             (None, 4)                 2052      
                                                                 
=================================================================
Total params: 3355204 (12.80 MB)
Trainable params: 3350980 (12.78 MB)
Non-trainable params: 4224 (16.50 KB)
_________________________________________________________________

# Evaluate the model on the test data
test_loss, test_accuracy = complex_cnn_model.evaluate(test_set_gray2, verbose=1)

# Print the results
print("Test Loss:", test_loss)
print("Test Accuracy:", test_accuracy)

4/4 [==============================] - 1s 158ms/step - loss: 1.2255 - accuracy: 0.4453
Test Loss: 1.225456714630127
Test Accuracy: 0.4453125

# Plotting the Training and Validation accuracies of EfficientNet model
plt.plot(history_complex_cnn.history['accuracy'])
plt.plot(history_complex_cnn.history['val_accuracy'])
plt.title('Complex CNN Model Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc = 'upper left')

# Display the plot
plt.show()

Observations and Insights:

• A significantly more complex custom CNN model has performed better than any of the transfer learning models trialed earlier, but it still hasn't performed as well in both training and validation than our custom CNN model for Greyscale images previously chosen.
  • Chosen CNN for Greyscale images: 71.1% training accuracy, 68% validation accuracy.
  • More complex CNN: 63.2% training accuracy, 64.1% validation accuracy.
• This more complex CNN model has taken significantly more processing power than its simpler counterpart, as expected. The unexpected outcome is its poorer performance, however.
• Again, training has stopped very prematurely due to Learning Rate platoeing, which might indicate early overfitting from a model that does not generalises enough for this task. Perhaps relaxing the conditions at which the condition is met might allow the model to improve its performance over time or could lead to further overfitting.

A significantly more complex custom CNN hasn't performed better than our selected CNN Model for Greyscale images. An early stopping was triggered at a very early epoch 13, which might have been due to the model not generalising enough for the task. Considering the above, the CNN Model for Greyscale images is still the best performing model thus far.

Plotting the Confusion Matrix for the chosen final model

# Plot the confusion matrix and generate a classification report for the model
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
test_set = datagen.flow_from_directory(folder_path + "test",
                                          target_size = (img_size[0],img_size[1]),
                                          color_mode = 'grayscale',
                                          batch_size = 128,
                                          class_mode = 'categorical',
                                          classes = ['happy', 'sad', 'neutral', 'surprise'],
                                          shuffle = True)
test_images, test_labels = next(test_set)

# Evaluating with model2_gray
pred = model2_gray.predict(test_images)
pred = np.argmax(pred, axis = 1)
y_true = np.argmax(test_labels, axis = 1)

# Printing the classification report
print(classification_report(y_true, pred, target_names=['happy', 'sad', 'neutral', 'surprise']))

# Plotting the heatmap using confusion matrix
cm = confusion_matrix(y_true, pred)
plt.figure(figsize = (8, 5))
sns.heatmap(cm, annot = True,  
            fmt = '.0f', 
            xticklabels = ['happy', 'sad', 'neutral', 'surprise'], 
            yticklabels = ['happy', 'sad', 'neutral', 'surprise'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()

Found 128 images belonging to 4 classes.
4/4 [==============================] - 0s 46ms/step
              precision    recall  f1-score   support

       happy       0.59      0.72      0.65        32
         sad       0.23      0.22      0.23        32
     neutral       0.30      0.28      0.29        32
    surprise       0.83      0.75      0.79        32

    accuracy                           0.49       128
   macro avg       0.49      0.49      0.49       128
weighted avg       0.49      0.49      0.49       128

Observations and Insights:

• Happy: This class shows a relatively high precision of 0.59 and a recall of 0.72, indicating that the model is better at correctly identifying 'happy' emotions and also capturing most of the 'happy' instances in the dataset.
• Sad: Both precision and recall are low for 'sad' emotions (0.23 precision, 0.22 recall), suggesting difficulties in accurately identifying and capturing all instances of sadness.
• Neutral: The 'neutral' class has a precision of 0.30 and a recall of 0.28, indicating a moderate ability to identify and capture neutral expressions, though with significant room for improvement.
• Surprise: This class has the highest precision (0.83) and a good recall rate (0.75), indicating a strong ability of the model to correctly identify 'surprise' emotions and capture most of the 'surprise' instances.

Trialing MobileNet V3 pre-trained model from Kaggle

import tensorflow_hub as hub
import torch
import torch.nn as nn
import torchvision.models as models
import torch.optim as optim
from torchvision import transforms
from torch.utils.data import DataLoader
from torchvision.datasets import ImageFolder

Trialing Data Augumentation

The idea is to apply various transformations, such as rotation, flipping, scaling, and changes in brightness or contrast, to the existing data. This process creates additional training samples, preventing the model from overfitting and enhancing its ability to generalize well to unseen data. Data augmentation is especially valuable when dealing with limited datasets, as it effectively expands the learning capacity of models, improving their robustness and overall performance.

# Create an instance of ImageDataGenerator with additional image transformations (Horizontal Flip and rotation)
datagen = ImageDataGenerator(
            rescale = 1. / 255,
            horizontal_flip = True,
            vertical_flip = False,
            rotation_range = 25,
            zoom_range=0.5,
            brightness_range=(0.,2.),
            shear_range=0.3)

MobileNet V3 uses 224x224 pixel input images, therefore a transformation for the input images should be done.

Fortunately, the ImageDataGenerator and flow_from_directory method in TensorFlow's Keras API can handle this resizing automatically, we just need to set the target_size parameter to (224, 224).

# Set the target size to 224x224
img_size = (224, 224)
print("Images will be scaled to " + str(img_size) + " pixels")

# Creating dataloader for train for RGB dataset
train_set_pre = datagen.flow_from_directory(folder_path + "train",
                                        target_size = (img_size),
                                        color_mode = 'rgb',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        classes = ['happy', 'sad', 'neutral', 'surprise'],
                                        shuffle = True)

# Creating dataloader for validation for RGB dataset
validation_set_pre = datagen.flow_from_directory(folder_path + "validation",
                                        target_size = (img_size),
                                        color_mode = 'rgb',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        classes = ['happy', 'sad', 'neutral', 'surprise'],
                                        shuffle = True)

# Creating dataloader for test for RGB dataset
test_set_pre = datagen.flow_from_directory(folder_path + "test",
                                        target_size = (img_size),
                                        color_mode = 'rgb',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        classes = ['happy', 'sad', 'neutral', 'surprise'],
                                        shuffle = True)

Images will be scaled to (224, 224) pixels
Found 15109 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.
Found 128 images belonging to 4 classes.

# Clearing backend
backend.clear_session()

# Reset seed for number generators using reset_seed() function
reset_seed(fixed_seed)

Random number generators seed has been set to 42

Starting a new experiment run in the specified "fine-tune mobilenetv3" project. All the training metrics, model parameters, outputs, and logs generated during this run can be tracked and visualized through the wandb dashboard. This is especially useful for comparing different runs, understanding model behavior, and iterating quickly during model development and fine-tuning.

# Importing Weights and Biases library and creating new project
"""
import wandb
from wandb.keras import WandbCallback
wandb.init(project="fine-tune mobilenetv3")
"""

'\nimport wandb\nfrom wandb.keras import WandbCallback\nwandb.init(project="fine-tune mobilenetv3")\n'

# mobilenetv3 will contain the imported model
mobilenetv3 = tf.keras.Sequential([
    hub.KerasLayer("https://www.kaggle.com/models/google/mobilenet-v3/frameworks/TensorFlow2/variations/large-075-224-feature-vector/versions/1",
                   trainable=False),  # Can be True, see below.
    tf.keras.layers.Dense(4, activation='softmax')
])
# Batch input shape with scaled images (from 48x48 to 224x224 pixels)
mobilenetv3.build([None, img_size[0], img_size[1], 3])  

mobilenetv3.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 keras_layer (KerasLayer)    (None, 1280)              2731616   
                                                                 
 dense (Dense)               (None, 4)                 5124      
                                                                 
=================================================================
Total params: 2736740 (10.44 MB)
Trainable params: 5124 (20.02 KB)
Non-trainable params: 2731616 (10.42 MB)
_________________________________________________________________

# Visualize the model as blocks
from tensorflow.keras.utils import plot_model
plot_model(mobilenetv3, to_file='model.png', show_shapes=True, show_layer_names=True)

# ModelCheckpoint is used to save the model at certain intervals
checkpoint_mobnetv3 = ModelCheckpoint("mobilenetv3.h5", # file path where the model will be saved
                    monitor='val_accuracy',  # means the callback monitors the validation accuracy
                    verbose=1,               # model is saved only when the metric (val_acc) has improved
                    save_best_only=True,
                    mode='max')

# Stop the training process early if a monitored metric has stopped improving
early_stopping = EarlyStopping(monitor = 'val_loss',  # monitors the validation loss
                          min_delta = 0,              # any positive change is considered as an improvement
                          patience = 3,               # training will be stopped if no improvement is seen for 3 epochs
                          verbose = 1,
                          restore_best_weights = True # weights are reverted to best value after early stopping
                          )

# Reduces the learning rate when a metric has stopped improving
reduce_learningrate = ReduceLROnPlateau(monitor = 'val_loss', # monitor the validation loss
                              factor = 0.2,                   # learning rate reduced to 20% when reduction is triggered
                              patience = 3,                   # reduction happens after 3 epochs with no improvement
                              verbose = 1,
                              min_delta = 0.0001)

# Configured callbacks will be passed to the model during training
callbacks_list_mobnetv3 = [early_stopping, checkpoint_mobnetv3, reduce_learningrate]

# Number of iterations over the entire dataset that the training process should run
epochs = 20

# Compiling the CNN model with categorical crossentropy as loss function, Adam Optimizer
# with 0.01 learning rate, and set metrics set to 'accuracy'.
mobilenetv3.compile(loss='categorical_crossentropy',
          optimizer=legacy.Adam(learning_rate = 0.01),
          metrics=['accuracy'])
total_sample=train_set_pre.n

# Fitting the model with required parameters above, saving model into history_resnet variable
history_mobilenetv3 = mobilenetv3.fit(
    train_set_pre,
    steps_per_epoch = train_set_pre.samples // train_set_pre.batch_size,
    validation_data = validation_set_pre,
    validation_steps = validation_set_pre.samples // validation_set_pre.batch_size,
    epochs = epochs,
    verbose = 1,
    callbacks = callbacks_list_mobnetv3
)

Epoch 1/20
472/472 [==============================] - ETA: 0s - loss: 1.4213 - accuracy: 0.4216
Epoch 1: val_accuracy improved from -inf to 0.42480, saving model to mobilenetv3.h5
472/472 [==============================] - 162s 339ms/step - loss: 1.4213 - accuracy: 0.4216 - val_loss: 1.4765 - val_accuracy: 0.4248 - lr: 0.0100
Epoch 2/20

/Users/ceb/anaconda3/lib/python3.11/site-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

472/472 [==============================] - ETA: 0s - loss: 1.3875 - accuracy: 0.4533
Epoch 2: val_accuracy improved from 0.42480 to 0.45101, saving model to mobilenetv3.h5
472/472 [==============================] - 191s 404ms/step - loss: 1.3875 - accuracy: 0.4533 - val_loss: 1.3379 - val_accuracy: 0.4510 - lr: 0.0100
Epoch 3/20
472/472 [==============================] - ETA: 0s - loss: 1.4253 - accuracy: 0.4483
Epoch 3: val_accuracy improved from 0.45101 to 0.48206, saving model to mobilenetv3.h5
472/472 [==============================] - 205s 434ms/step - loss: 1.4253 - accuracy: 0.4483 - val_loss: 1.3701 - val_accuracy: 0.4821 - lr: 0.0100
Epoch 4/20
472/472 [==============================] - ETA: 0s - loss: 1.3845 - accuracy: 0.4584
Epoch 4: val_accuracy improved from 0.48206 to 0.48831, saving model to mobilenetv3.h5
472/472 [==============================] - 193s 408ms/step - loss: 1.3845 - accuracy: 0.4584 - val_loss: 1.3222 - val_accuracy: 0.4883 - lr: 0.0100
Epoch 5/20
472/472 [==============================] - ETA: 0s - loss: 1.3962 - accuracy: 0.4552
Epoch 5: val_accuracy did not improve from 0.48831
472/472 [==============================] - 208s 441ms/step - loss: 1.3962 - accuracy: 0.4552 - val_loss: 1.5967 - val_accuracy: 0.4006 - lr: 0.0100
Epoch 6/20
472/472 [==============================] - ETA: 0s - loss: 1.3959 - accuracy: 0.4589
Epoch 6: val_accuracy improved from 0.48831 to 0.49516, saving model to mobilenetv3.h5
472/472 [==============================] - 162s 343ms/step - loss: 1.3959 - accuracy: 0.4589 - val_loss: 1.3347 - val_accuracy: 0.4952 - lr: 0.0100
Epoch 7/20
472/472 [==============================] - ETA: 0s - loss: 1.4257 - accuracy: 0.4575Restoring model weights from the end of the best epoch: 4.

Epoch 7: val_accuracy did not improve from 0.49516

Epoch 7: ReduceLROnPlateau reducing learning rate to 0.0019999999552965165.
472/472 [==============================] - 173s 367ms/step - loss: 1.4257 - accuracy: 0.4575 - val_loss: 1.3266 - val_accuracy: 0.4887 - lr: 0.0100
Epoch 7: early stopping

Custom Classifier with MobileNetV3 Feature Extractor

# Clearing backend
backend.clear_session()

# Reset seed for number generators using reset_seed() function
reset_seed(fixed_seed)

Random number generators seed has been set to 42

# Define MobileNetV3 as a feature extractor
mobilenet_v3 = hub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v3_large_075_224/feature_vector/5",
                              trainable=False)

mnetv3_2 = tf.keras.Sequential([
    # Feature extractor
    mobilenet_v3,
    
    # Adding Dense layer with 128 nodes and "relu" activation
    tf.keras.layers.Dense(128, activation="relu"),
    # Adding dropout layer 
    tf.keras.layers.Dropout(0.5),
    # BatchNormalisation layer
    tf.keras.layers.BatchNormalization(),
    # Flattening all layers
    tf.keras.layers.Flatten(),
    # Output layer with 4 classes and "softmax" activation
    tf.keras.layers.Dense(4, activation="softmax")  
])

# Batch input shape with scaled images (from 48x48 to 224x224 pixels)
mnetv3_2.build([None, img_size[0], img_size[1], 3])  

# Define MobileNetV3 as a feature extractor
mobilenet_v3 = hub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v3_large_075_224/feature_vector/5",
                              trainable=False,input_shape=(224,224,3))

SEBlock: Squeeze-and-Excitation Block for Improved Feature Representation

The "SEBlock" is a Squeeze-and-Excitation block designed to enhance feature representation in neural networks. By dynamically recalibrating channel-wise feature responses, it improves the model's ability to capture important patterns and boost overall performance. The block consists of a Global Average Pooling layer followed by two fully connected layers, contributing to a more adaptive and effective feature learning process.

class SEBlock(tf.keras.layers.Layer):
    def __init__(self, ratio=16):
        super(SEBlock, self).__init__()
        self.ratio = ratio
        # Removing the GlobalAveragePooling2D layer since it's not needed for 2D inputs

    def build(self, input_shape):
        num_channels = input_shape[-1]
        self.fc1 = tf.keras.layers.Dense(num_channels // self.ratio, activation='relu')
        self.fc2 = tf.keras.layers.Dense(num_channels, activation='sigmoid')

    def call(self, inputs):
        # Using `inputs` instead of `x`
        x = self.fc1(inputs)
        x = self.fc2(x)
        return x * inputs

# Create and build the model
target_size = (224, 224)
model = tf.keras.Sequential([
    mobilenet_v3,
    SEBlock(ratio=16),
    tf.keras.layers.Dense(128, activation="relu"),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(4, activation="softmax")
])

model.build((32, *target_size, 3))  # Define the input shape

# Compiling the CNN model with categorical crossentropy as loss function, Adam Optimizer
# with 0.01 learning rate, and set metrics set to 'accuracy'.
model.compile(loss='categorical_crossentropy',
          optimizer=legacy.Adam(learning_rate = 0.01),
          metrics=['accuracy'])
total_sample=train_set_pre.n

#Model summary
mnetv3_2.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 keras_layer (KerasLayer)    (None, 1280)              2731616   
                                                                 
 dense (Dense)               (None, 128)               163968    
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 batch_normalization (Batch  (None, 128)               512       
 Normalization)                                                  
                                                                 
 flatten (Flatten)           (None, 128)               0         
                                                                 
 dense_1 (Dense)             (None, 4)                 516       
                                                                 
=================================================================
Total params: 2896612 (11.05 MB)
Trainable params: 164740 (643.52 KB)
Non-trainable params: 2731872 (10.42 MB)
_________________________________________________________________

# ModelCheckpoint is used to save the model at certain intervals
checkpoint_mobnetv3_1 = ModelCheckpoint("mobilenetv3_1.h5", # file path where the model will be saved
                    monitor='val_accuracy',  # means the callback monitors the validation accuracy
                    verbose=1,               # model is saved only when the metric (val_acc) has improved
                    save_best_only=True,
                    mode='max')

# Configured callbacks will be passed to the model during training
callbacks_list_mobnetv3 = [early_stopping, checkpoint_mobnetv3_1, reduce_learningrate]

# Number of iterations over the entire dataset that the training process should run
epochs = 20

# Fitting the model with required parameters above, saving model into history_resnet variable
history_mobilenetv3_1 = model.fit(
    train_set_pre,
    steps_per_epoch = train_set_pre.samples // train_set_pre.batch_size,
    validation_data = validation_set_pre,
    validation_steps = validation_set_pre.samples // validation_set_pre.batch_size,
    epochs = epochs,
    verbose = 1,
    callbacks = callbacks_list_mobnetv3
)

Epoch 1/20
472/472 [==============================] - ETA: 0s - loss: 1.3299 - accuracy: 0.3697
Epoch 1: val_accuracy improved from -inf to 0.43871, saving model to mobilenetv3_1.h5
472/472 [==============================] - 167s 352ms/step - loss: 1.3299 - accuracy: 0.3697 - val_loss: 1.2176 - val_accuracy: 0.4387 - lr: 0.0100
Epoch 2/20
472/472 [==============================] - ETA: 0s - loss: 1.2501 - accuracy: 0.4137
Epoch 2: val_accuracy improved from 0.43871 to 0.44375, saving model to mobilenetv3_1.h5
472/472 [==============================] - 160s 339ms/step - loss: 1.2501 - accuracy: 0.4137 - val_loss: 1.2040 - val_accuracy: 0.4437 - lr: 0.0100
Epoch 3/20
472/472 [==============================] - ETA: 0s - loss: 1.2461 - accuracy: 0.4222
Epoch 3: val_accuracy did not improve from 0.44375
472/472 [==============================] - 164s 347ms/step - loss: 1.2461 - accuracy: 0.4222 - val_loss: 1.2318 - val_accuracy: 0.4187 - lr: 0.0100
Epoch 4/20
472/472 [==============================] - ETA: 0s - loss: 1.2290 - accuracy: 0.4295
Epoch 4: val_accuracy did not improve from 0.44375
472/472 [==============================] - 196s 414ms/step - loss: 1.2290 - accuracy: 0.4295 - val_loss: 1.1923 - val_accuracy: 0.4347 - lr: 0.0100
Epoch 5/20
472/472 [==============================] - ETA: 0s - loss: 1.2213 - accuracy: 0.4389
Epoch 5: val_accuracy improved from 0.44375 to 0.47661, saving model to mobilenetv3_1.h5
472/472 [==============================] - 184s 391ms/step - loss: 1.2213 - accuracy: 0.4389 - val_loss: 1.1653 - val_accuracy: 0.4766 - lr: 0.0100
Epoch 6/20
472/472 [==============================] - ETA: 0s - loss: 1.2211 - accuracy: 0.4378
Epoch 6: val_accuracy did not improve from 0.47661
472/472 [==============================] - 172s 364ms/step - loss: 1.2211 - accuracy: 0.4378 - val_loss: 1.2145 - val_accuracy: 0.4302 - lr: 0.0100
Epoch 7/20
472/472 [==============================] - ETA: 0s - loss: 1.2170 - accuracy: 0.4377
Epoch 7: val_accuracy did not improve from 0.47661
472/472 [==============================] - 167s 354ms/step - loss: 1.2170 - accuracy: 0.4377 - val_loss: 1.1858 - val_accuracy: 0.4581 - lr: 0.0100
Epoch 8/20
472/472 [==============================] - ETA: 0s - loss: 1.2119 - accuracy: 0.4498Restoring model weights from the end of the best epoch: 5.

Epoch 8: val_accuracy did not improve from 0.47661

Epoch 8: ReduceLROnPlateau reducing learning rate to 0.0019999999552965165.
472/472 [==============================] - 186s 394ms/step - loss: 1.2119 - accuracy: 0.4498 - val_loss: 1.2090 - val_accuracy: 0.4480 - lr: 0.0100
Epoch 8: early stopping

Removing early stopping on learning rate plateau

# ModelCheckpoint is used to save the model at certain intervals
checkpoint_mobnetv3_2 = ModelCheckpoint("mobilenetv3_2.h5", # file path where the model will be saved
                    monitor='val_accuracy',  # means the callback monitors the validation accuracy
                    verbose=1,               # model is saved only when the metric (val_acc) has improved
                    save_best_only=True,
                    mode='max')

# Configured callbacks will be passed to the model during training
callbacks_list_mobnetv3_2 = [early_stopping, checkpoint_mobnetv3_2]#, reduce_learningrate]

# Number of iterations over the entire dataset that the training process should run
epochs = 20

# Fitting the model with required parameters above, saving model into history_resnet variable
history_mobilenetv3_2 = model.fit(
    train_set_pre,
    steps_per_epoch = train_set_pre.samples // train_set_pre.batch_size,
    validation_data = validation_set_pre,
    validation_steps = validation_set_pre.samples // validation_set_pre.batch_size,
    epochs = epochs,
    verbose = 1,
    callbacks = callbacks_list_mobnetv3_2
)

Epoch 1/20
472/472 [==============================] - ETA: 0s - loss: 1.1997 - accuracy: 0.4504
Epoch 1: val_accuracy improved from -inf to 0.47500, saving model to mobilenetv3_2.h5
472/472 [==============================] - 167s 354ms/step - loss: 1.1997 - accuracy: 0.4504 - val_loss: 1.1579 - val_accuracy: 0.4750
Epoch 2/20
472/472 [==============================] - ETA: 0s - loss: 1.1946 - accuracy: 0.4586
Epoch 2: val_accuracy improved from 0.47500 to 0.47964, saving model to mobilenetv3_2.h5
472/472 [==============================] - 178s 378ms/step - loss: 1.1946 - accuracy: 0.4586 - val_loss: 1.1509 - val_accuracy: 0.4796
Epoch 3/20
472/472 [==============================] - ETA: 0s - loss: 1.1850 - accuracy: 0.4665
Epoch 3: val_accuracy did not improve from 0.47964
472/472 [==============================] - 207s 439ms/step - loss: 1.1850 - accuracy: 0.4665 - val_loss: 1.1576 - val_accuracy: 0.4778
Epoch 4/20
472/472 [==============================] - ETA: 0s - loss: 1.1855 - accuracy: 0.4688
Epoch 4: val_accuracy improved from 0.47964 to 0.49738, saving model to mobilenetv3_2.h5
472/472 [==============================] - 192s 408ms/step - loss: 1.1855 - accuracy: 0.4688 - val_loss: 1.1376 - val_accuracy: 0.4974
Epoch 5/20
472/472 [==============================] - ETA: 0s - loss: 1.1857 - accuracy: 0.4657
Epoch 5: val_accuracy did not improve from 0.49738
472/472 [==============================] - 168s 354ms/step - loss: 1.1857 - accuracy: 0.4657 - val_loss: 1.1407 - val_accuracy: 0.4891
Epoch 6/20
472/472 [==============================] - ETA: 0s - loss: 1.1822 - accuracy: 0.4709
Epoch 6: val_accuracy did not improve from 0.49738
472/472 [==============================] - 158s 334ms/step - loss: 1.1822 - accuracy: 0.4709 - val_loss: 1.1483 - val_accuracy: 0.4879
Epoch 7/20
472/472 [==============================] - ETA: 0s - loss: 1.1791 - accuracy: 0.4686Restoring model weights from the end of the best epoch: 4.

Epoch 7: val_accuracy did not improve from 0.49738
472/472 [==============================] - 150s 317ms/step - loss: 1.1791 - accuracy: 0.4686 - val_loss: 1.1406 - val_accuracy: 0.4879
Epoch 7: early stopping

Evaluating the MobileNetV3 Model from Kaggle

import tensorflow as tf
import tensorflow_hub as hub

# Ensure that the SEBlock class is defined exactly as it was when the model was saved
class SEBlock(tf.keras.layers.Layer):
    def __init__(self, ratio=16):
        super(SEBlock, self).__init__()
        self.ratio = ratio
        # Removing the GlobalAveragePooling2D layer since it's not needed for 2D inputs

    def build(self, input_shape):
        num_channels = input_shape[-1]
        self.fc1 = tf.keras.layers.Dense(num_channels // self.ratio, activation='relu')
        self.fc2 = tf.keras.layers.Dense(num_channels, activation='sigmoid')

    def call(self, inputs):
        # Using `inputs` instead of `x`
        x = self.fc1(inputs)
        x = self.fc2(x)
        return x * inputs

# Load the model with custom object scope
with tf.keras.utils.custom_object_scope({'KerasLayer': hub.KerasLayer, 'SEBlock': SEBlock}):
    mobilenetv3_2 = tf.keras.models.load_model("mobilenetv3_2.h5")

# Evaluate the model on the test data
test_loss, test_accuracy = mobilenetv3_2.evaluate(test_set_pre, verbose=1)

# Print the results
print("Test Loss:", test_loss)
print("Test Accuracy:", test_accuracy)

4/4 [==============================] - 2s 219ms/step - loss: 1.0917 - accuracy: 0.5078
Test Loss: 1.0916602611541748
Test Accuracy: 0.5078125

Trying out best CNN model with data augmentation (Image Rotation and Brightness changes)

Trialing Data Augumentation

The idea is to apply various transformations, such as rotation, flipping, scaling, and changes in brightness or contrast, to the existing data. This process creates additional training samples, preventing the model from overfitting and enhancing its ability to generalize well to unseen data. Data augmentation is especially valuable when dealing with limited datasets, as it effectively expands the learning capacity of models, improving their robustness and overall performance.

# Create an instance of ImageDataGenerator with additional image transformations (Horizontal Flip and rotation)
datagen = ImageDataGenerator(
            rescale = 1. / 255,
            horizontal_flip = True,
            vertical_flip = False,
            width_shift_range=0.2,
            height_shift_range=0.2,
            rotation_range = 25,
            zoom_range=0.5,
            brightness_range=(0.,2.),
            shear_range=0.3,
            fill_mode='nearest')    

# Set the target size to 48x48 again
img_size = (48, 48)

# Creating dataloader for train for Grayscale dataset
train_set_gray = datagen.flow_from_directory(folder_path + "train",
                                        target_size = (img_size),
                                        color_mode = 'grayscale',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        shuffle = True)

# Creating dataloader for validation for Grayscale dataset
validation_set_gray = datagen.flow_from_directory(folder_path + "validation",
                                        target_size = (img_size),
                                        color_mode = 'grayscale',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        shuffle = True)

# Creating dataloader for test for Grayscale dataset
test_set_gray = datagen.flow_from_directory(folder_path + "test",
                                        target_size = (img_size),
                                        color_mode = 'grayscale',
                                        batch_size = batch_size,
                                        class_mode = 'categorical',
                                        shuffle = True)

Found 15109 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.
Found 128 images belonging to 4 classes.

# Clearing backend
backend.clear_session()

# Reset seed for number generators using reset_seed() function
reset_seed(fixed_seed)

Random number generators seed has been set to 42

# Load the saved model if notebook session gets disconnected
model2_gray_da = load_model("model2_grayscale.h5")

# ModelCheckpoint is used to save the model at certain intervals
checkpoint_best_da = ModelCheckpoint("best_da.h5", # file path where the model will be saved
                    monitor='val_accuracy',  # means the callback monitors the validation accuracy
                    verbose=1,               # model is saved only when the metric (val_acc) has improved
                    save_best_only=True,
                    mode='max')

# Stop the training process early if a monitored metric has stopped improving
early_stopping = EarlyStopping(monitor = 'val_loss',  # monitors the validation loss
                          min_delta = 0,              # any positive change is considered as an improvement
                          patience = 3,               # training will be stopped if no improvement is seen for 3 epochs
                          verbose = 1,
                          restore_best_weights = True # weights are reverted to best value after early stopping
                          )

# Configured callbacks will be passed to the model during training
callbacks_list_mobnetv3 = [early_stopping, checkpoint_best_da]#, reduce_learningrate]

# Number of iterations over the entire dataset that the training process should run
epochs = 20

# Train the model
history = model2_gray_da.fit(
    train_set_gray,
    steps_per_epoch=train_set_gray.samples // train_set_gray.batch_size,
    validation_data=validation_set_gray,
    validation_steps=validation_set_gray.samples // validation_set_gray.batch_size,
    epochs=epochs
)

Epoch 1/20
472/472 [==============================] - 100s 210ms/step - loss: 1.1809 - accuracy: 0.4630 - val_loss: 1.1547 - val_accuracy: 0.4879
Epoch 2/20
472/472 [==============================] - 108s 229ms/step - loss: 1.1257 - accuracy: 0.5037 - val_loss: 1.2045 - val_accuracy: 0.4333
Epoch 3/20
472/472 [==============================] - 132s 280ms/step - loss: 1.1144 - accuracy: 0.5008 - val_loss: 1.1813 - val_accuracy: 0.4460
Epoch 4/20
472/472 [==============================] - 102s 216ms/step - loss: 1.0910 - accuracy: 0.5213 - val_loss: 1.0842 - val_accuracy: 0.5032
Epoch 5/20
472/472 [==============================] - 142s 302ms/step - loss: 1.0754 - accuracy: 0.5246 - val_loss: 1.1057 - val_accuracy: 0.4990
Epoch 6/20
472/472 [==============================] - 139s 295ms/step - loss: 1.0696 - accuracy: 0.5228 - val_loss: 1.1310 - val_accuracy: 0.4821
Epoch 7/20
472/472 [==============================] - 113s 240ms/step - loss: 1.0529 - accuracy: 0.5360 - val_loss: 1.0399 - val_accuracy: 0.5427
Epoch 8/20
472/472 [==============================] - 116s 246ms/step - loss: 1.0553 - accuracy: 0.5380 - val_loss: 1.0099 - val_accuracy: 0.5579
Epoch 9/20
472/472 [==============================] - 97s 206ms/step - loss: 1.0425 - accuracy: 0.5439 - val_loss: 1.1024 - val_accuracy: 0.5050
Epoch 10/20
472/472 [==============================] - 90s 190ms/step - loss: 1.0370 - accuracy: 0.5496 - val_loss: 1.0711 - val_accuracy: 0.5125
Epoch 11/20
472/472 [==============================] - 96s 204ms/step - loss: 1.0268 - accuracy: 0.5565 - val_loss: 1.0818 - val_accuracy: 0.5095
Epoch 12/20
472/472 [==============================] - 101s 214ms/step - loss: 1.0181 - accuracy: 0.5556 - val_loss: 1.0240 - val_accuracy: 0.5498
Epoch 13/20
472/472 [==============================] - 109s 231ms/step - loss: 1.0078 - accuracy: 0.5617 - val_loss: 1.0130 - val_accuracy: 0.5466
Epoch 14/20
472/472 [==============================] - 98s 207ms/step - loss: 1.0102 - accuracy: 0.5618 - val_loss: 1.0570 - val_accuracy: 0.5373
Epoch 15/20
472/472 [==============================] - 102s 217ms/step - loss: 1.0057 - accuracy: 0.5578 - val_loss: 1.0006 - val_accuracy: 0.5603
Epoch 16/20
472/472 [==============================] - 108s 229ms/step - loss: 1.0074 - accuracy: 0.5594 - val_loss: 1.0061 - val_accuracy: 0.5599
Epoch 17/20
472/472 [==============================] - 104s 220ms/step - loss: 1.0051 - accuracy: 0.5643 - val_loss: 0.9993 - val_accuracy: 0.5756
Epoch 18/20
472/472 [==============================] - 123s 261ms/step - loss: 0.9910 - accuracy: 0.5707 - val_loss: 1.0433 - val_accuracy: 0.5317
Epoch 19/20
472/472 [==============================] - 120s 253ms/step - loss: 0.9944 - accuracy: 0.5705 - val_loss: 0.9757 - val_accuracy: 0.5802
Epoch 20/20
472/472 [==============================] - 108s 228ms/step - loss: 0.9810 - accuracy: 0.5748 - val_loss: 1.0382 - val_accuracy: 0.5367

# Evaluate the model on the test data using Grayscale model
test_loss, test_accuracy = model2_gray_da.evaluate(test_set_gray, verbose=1)

# Print the results
print("Test Loss:", test_loss)
print("Test Accuracy:", test_accuracy)

4/4 [==============================] - 0s 80ms/step - loss: 1.0238 - accuracy: 0.5703
Test Loss: 1.0238081216812134
Test Accuracy: 0.5703125

# Plotting the Training and Validation accuracies of CNN model 2 and Grayscale images
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('Best CNN Model with Data Augmentation')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc = 'upper left')

# Display the plot
plt.show()

Observations and Insights:

• Surprisingly, in trying to improve the model by doing data augmentation has actually caused a deterioration in performance.
  • Chosen CNN with data augmentation: 57.5% training accuracy, 53.7% validation accuracy.

Considering the above, the CNN Model for Greyscale images without data augmentation is the best performing model trialed, therefore the selected model.

Conclusion:

EDA Observations:

• The provided images were all of the same size 48x48 pixels. From the images sampled, all images seem to be useful.
• Images provided in the dataset are Greyscale (monochromatic), which should be taken into consideration by the ML Engineer.
• There is an under-representation of the "surprise" class in the training and validation datasets, which in a first instance have raised a suspicion that this class would under-perform.
• Expressions of distinct classes are very similar, for instance between sad and neutral expressions. These images, if unlabeled, would be tricky to be classified even for a human eye.
• Several of the images have watermarks, which would potentially affect the outcome of the models' training and predictions.

CNNs and Transfer Learning model trialing:

• All of the Transfer Leaning models trialed have stopped early due to a plateau in the Learning Rate, which likely indicates that these models do not generalise enough for this particular task.
  • VGG16: 50.2% training accuracy, 47.4% validation accuracy.
  • ResnetV2: 38.2% training accuracy, 30.75% validation accuracy.
  • EfficientNet: 26.1% training accuracy, 26% validation accuracy.

MobileNet V3 model from Kaggle has also been trialed, with data augmentation and removing the early stopping on Learning Rate plateau, but still hasn't performed as well as the custom CNN for Greyscale images:

• MobileNet V3: 46.8% training accuracy, 48.8% validation accuracy.

The best CNN trialed has provided a 71.1% training accuracy, 68% validation accuracy. Surprisingly, Data augmentation was also performed to input data to best performing CNN trialed in this notebook, but it has caused a deterioration in the model's performance (Training and Validation accuracies dropped to 57% and 53% respectively).

• In general, utilising RGB images as the input to CNNs do not improve the performance. This is perhaps expected, since the images provided in the dataset are all in greyscale.
• The custom CNN model for grayscale images significantly outperforms the transfer learning models (VGG16, ResnetV2, and EfficientNet) in both training and validation accuracies.
• Increasing the complexity of the custom CNN did not result in a better performance
  • More complex CNN: 63.2% training accuracy, 64.1% validation accuracy.

The selected model was a "medium" complexity custom CNN model, that seems to generalise enough for the task in hands with 71.1% training accuracy and 68% validation accuracy and a very reasonable processing time.

More complex pre-trained models did not perform as well as our custom "medium" comnplexity CNN model.

Although the "surprise" class was the most imbalanced class dataset, with 20% less training and validation data, the real surprise was with the results of the confusion matrix with the test data where it has outperformed all other classes.

- "surprise" had the highest precision (0.83) and a good recall rate (0.75).
- "happy" had the second best performance with a precision of 0.59 and a recall of 0.72.
- "neutral" class stands in third, with a precision of 0.30 and a recall of 0.28
- "sad" has performed the worst, with 0.23 precision, 0.22 recall

This is perhaps due to "surprise" expressions being very clear with features such as an open mouth, present in most images. It is also very tricky for the human eye to distinguish between some of the "neutral" and "sad" expressions, possibly the reason for the poor performance of these two classes.

Future Improvement Suggestions:

• Learning how to generalise more the Transfer Learning models, which in theory should perform better than the other models
• Providing more training data could potentially improve the accuracy of the models, especially for the "happy" and "surprised" datasets, which can cause confusion in the training.
• Reducing the imbalance between the different classes will most likely improve the results.
• Providing actual RGB images might also improve the classification precision, since colour data might reveal subtle differences in emotions, such as shadowing and eye colour when subjects are smalling and with eyes wide open.